Post on 21-Apr-2017
transcript
WELCOME TO ANALYTICS ACADEMY!
A/B Testing and Creating a Culture of Experimentation
A/B experiments show users different versions of your site and then compare results
TEST FIRST: FAST
● Can often mock up a feature in the testing tool first, without involving a tech queue
● Measuring results isn’t affected by seasonality, or other marketing efforts, or changes to the consumer mood because you are testing one group randomly divided, so all these factors are controlled for
● Results are statistically tested and validated
BUILD FIRST: SLOW
● Higher up-front investment: Have to invest in building the feature without knowing if it will work
● Hard to measure results: you roll out the feature and compare conversion before (5-7% for last two months) to after (5.5-6.7% for month after). Is this an improvement or normal variation? Is it affected by seasonality? Did the email campaign that went out last week affect this rate?
Benefits of A/B testing
Experiment 1: Tools in StoreHypothesis: ● Replacing "Tools" with a more
learning-centered phrase will produce more click throughs
What’s the result?
Experiment 1: Tools in StoreHypothesis: ● Replacing "Tools" with a more
learning-centered phrase will produce more click throughs
What’s the result?● Learning Tools click rate up 16%● Teach Yourself click rate up 27%
Hypothesis: ● Changing to one year would have
no negative effect on subscription conversion
What’s the result?●
What we think●
Experiment 2: One year vs 10 issues
Hypothesis: ● Changing to one year would have
no negative effect on subscription conversion
What’s the result?● One year variation subscription
rate up 9%
What we think● People understand the value of a
year more than number of issues
Experiment 2: One year vs 10 issues
Experiment 3: Formatting on product pageHypothesis: ● Improving formatting would
increase product purchase
What’s the result?●
What we think●
Experiment 3: Formatting on product pageHypothesis: ● Improving formatting would
increase product purchase
What’s the result?● No statistical difference in
product purchase
What we think● Was formatted description too
long? Should we have short text and preview page?
A/B testing can inspire cultural change● Practice with A/B tests builds experimentation muscles
○ People practice the steps to build a good experiment so they start to feel obvious
○ A/B tests require good methodology: you are forced to pick a goal to measure; you automatically have a control group; the software collects and reports on the results
○ The benefits of the speed/clarity from these experiments increase demand for similar speed/clarity in areas outside the website
Ideal state for all business stakeholders for all questions: always ask, “Can this be an experiment?”
Basic human nature makes this hard
● Short term, it feels easier to make a decision based on gut feel, or defer to highest paid person’s opinion (HiPPO), or just try something and see what happens without formalizing a hypothesis or measuring the result (but still call it an experiment)
What makes it hard to experiment?
In the long run, it is actually a LOT easier to run an experiment
● Fail fast: have an idea? Experiment with a minimum viable product to see if the idea deserves further development--or not
● Decisions vs more discussion: The organization can move a lot faster when there’s certainty around a course of action. When there’s uncertainty, healthy discussion can sometimes sour into multiple meetings and prolonged debates, or, perhaps worse, unspoken doubts sap the momentum for the group moving forward
Experimentation done well becomes self-reinforcing, as people see how much easier/faster they can work
A/B testing shows experiments are easier
● New product development: set out hypotheses about the market (e.g. “managers want to buy HBP materials to help their direct reports”) and then test with customers (e.g. customer interviews where we learn that there’s an equally large market from coaches). Key is to build and test in stages, so you validate hypotheses along the way
● Email testing: split list as randomly as possible and send different emails to the two groups
● Before and after testing: Create an insider newsletter and compare subscriber engagement before and after
● A/B testing: use a formal A/B testing platform on the website
Some ways that HBR experiments
Change is as good as a rest: Sometimes a change tests well just because it is a change and gets people’s attention. Change a button from red to blue, you may get higher click throughs; effect diminishes over time, then six months later change it back to red and get higher click throughs
Focus on the big picture: don’t look at a change in isolation; look at the total impact. Adding a newsletter widget that gets clicks is good, but does it increase the total newsletter signups (or just cannibalize the clicks you are getting from other widgets)? Does the additional visual clutter lower overall engagement (higher bounce, lower time on site)?
Follow some best practices
But remember we are not a labHaving a culture of experimentation does not mean that your group transforms into a medical lab where we need 98% certainty and huge sample sizes to make a decision
Perfect is the enemy of the good: It’s better to have 20 good experiments than 3 perfect ones
Hurdle: is the experimental results better information that what you would have used otherwise? (e.g. better than gut instinct?)
Questions?
ResourcesGoogle analytics experiments: FREE! (but anecdotally pretty hard to use)
Optimizely: easy interface, great training resources to help you get acquainted with testing (we started here)
VWO: may be cheaper than Optimizely
Adobe Target: more robust, integrates with Adobe Analytics in a very powerful way (we moved here in January)
Facebook Ads:Audiences and ImpactJoseph Casciano, Harvard Public Affairs and Communications
How much to spend?
Why are we spending it?
● Low-value objective● Larger audience
● High-value objective● Smaller audience
Problems with Interest Targeting
● Often inaccurate, so serves your ad to the wrong people.
● Hard to get a precise, smaller, well-targeted audience.
● To big audiences, Facebook serves your ad to whoever is cheap and easy.
Business Manager
↓
Assets
↓
Audiences
↓
Create Audience
↓
Custom Audience
Metric Time
Relevance Score
CPM (cost per 1,000 impressions)
Results and Reach
Custom audiences make metrics matter
● Don’t provide contextless numbers about faceless masses.
● Tell concrete, true stories about your valued audience.
● So: “We reached half of our email subscribers on Facebook, half of who watched the video for more than three seconds.”
● Not: “We reached 132,674 Malaysian bots who couldn’t even theoretically fly into Cambridge for our symposium.”
● Use custom audiences○ To guide your budget○ To make metrics matter
Questions?
joseph_casciano@harvard.edu
Google Analytics Tips Tricks
Elizabeth Brady, EWB Analytics
IntroElizabeth Brady, Founder & Principal Web Analyst - EWB Analytics LLC - launched
March 2010
Specialties: Google Analytics and Google Tag Manager implementations, site audits,
web analytics support during site re-launch, measurement strategy and ongoing
analysis
Harvard groups I have collaborated with since 2012: Digital Communications, Harvard
Alumni, Harvard Admissions, Harvard Innovation Lab, Kennedy School, Harvard
Library, Harvard Learning Portal, HWPI, Ash Center
Contact: elizabeth@ewbanalytics.com
Keep It Clean
Filters Can Help PreventInternal Traffic
m 1
Maintain ‘exclude’ filter of
known internal IP
addresses
Ghost Spam
Maintain ‘include’ filter of
valid site hostnames
Crawler Spam
Maintain ‘exclude’ filter of
list of known spam
referrers
Filter: Exclude Internal Traffic by IP AddressInternal traffic inflates conversions & conversion rates. Check
current IP address by visiting whatismyip.com
IP AddressesUse regular expressions for a range of IP
addresses (ask IT for office IP range)
Dynamic IP Addresses
(residential)
Verify/update regularly
Test ActivityUse custom dimensions to track test users
even when not on internal network
Filter: Test Accounts by Custom Dimension● Use a custom dimension to
identify a test visitor who
visits a specific internal page
(webadmin, test, etc)
● Set the custom dimension at
the ‘user’ level
● Create a filter for any traffic
with that custom dimension
value
Ghost referrer spam:● Never actually visits your site● Sends data via the ‘measurement profile’
randomly to your GA account (became an issue only with Universal Analytics)
● Sends data with a missing (not set) or inaccurate hostname
● Can be prevented with a valid ‘hostname’ (include) filter
Prevent Ghost Referrer Spam
Filter: Valid Hostname(s)
Crawler spam:● Actually crawls/visits your site so the traffic appears legitimate● Filter this traffic by filtering on ‘campaign source’● Sample ‘exclude’ filter for known spam crawlers and domains
referenced as referrals from spam crawlers:semalt|anticrawler|best-seo-offer|best-seo-solution|buttons-for-website|buttons-for-your-website|7makemoneyonline|-musicas*-grat
is|kambasoft|savetubevideo|ranksonic|medispainstitute|offers.bycontext|100dollars-seo|sitevaluation|dailyrank
● Full set of 4 filters for crawler spam can be found here:http://help.analyticsedge.com/spam-filter/definitive-guide-to-removing-google-analytics-spam/
Filter: Crawler Spam
Language Spam - New Spam in 2016Language Spam
● Rather than referrers, the spamming
sites inserted spam messages into the
‘language’ reports
● Most do not use valid hostnames so
this would also be prevented with a
‘hostname’ include filter
● Additional exclude filters can be added
to address language spam
Lowercase Filters● Especially when starting a new
view, lowercase filters can avoid
capitalization inconsistencies
● Recommended for - page,
campaign
(medium/source/campaign),
search term (on site search)
Query String Cleanup● Google Analytics includes any query string
parameters (after the ‘?’ as part of the URL)
● Leads to multiple versions of the same ‘page’
and a challenge aggregating data
● Parameters to exclude can be identified in a
list in the view settings, or you could ‘go
nuclear’ and exclude them all with this filter
on the right
● Full URL with query strings (or just query
strings) can be captured as a custom
dimension to be viewed when needed
Check Your Setup
Referrer Exclusion List (Property Level)● Make sure your site subdomain/s is
included (new properties set up with
Universal Analytics will have this set
up on creation but any older site rolled
over to Universal Analytics did not
automatically have this configured)
● Include any off-site flows (login
validation, back-end sites like
pin1.harvard.edu) to prevent triggering
a new session
● Do not set up harvard.edu in the
exclusion list (that will prevent any
other Harvard sites showing up as
‘referrals’) - they will be ‘direct traffic’
Check ‘Exclude Bots & Spiders’ (View)● Excludes traffic from sites on the IAB
(Interactive Advertising Bureau) list of
known bots & spiders
● Sometimes these can be contracted
services like site response time (like
Gomez) that execute javascript and
would otherwise show up in reports
● It is recommended to leave this
unchecked for the unfiltered view
Link Google Search Console (Property) for Organic Search Trends
● Google Analytics no longer has much
insight into Google organic search
keywords
● Link your site’s Search Console
(formerly ‘Webmaster Tools’) account
for impressions/clicks on Google
Remember Campaign ‘Timeout’ (Property) is Configurable
● Standard campaign setting is 6 months
(sessions and conversions will be
credited to the last campaign in the
past 6 months)
● This explains why you may see
traffic/conversion for ‘old’ campaigns
● Your business group may decide you
need a shorter or longer campaign
timeout
Data Import (File Upload) Can Extend Analysis
● Data import lets you append
data to any dimension (standard
or custom) you collect
● The actual import can be a
simple text file upload
● Some uses might be to add
details around campaigns, add
authors or other details to
content pages, or group
information differently than
they way it is grouped in Google
Analytics
Reporting & Analysis
Custom Reports
Don’t dig for your data!
404s
Social Media Details
Top Pages by Type
Top Events by Type
Deep dive into a certain source of
traffic (ex: email campaigns)
Tracking 404’s
● No extra tagging needed
● Report Filter: Page title contains ‘Page Not Found’
● Dimensions: URL (page requested), Previous Page (might be entrance),
source/medium (more important for entrance pages to understand source of
traffic)
404 Error Report
● Monitor 404 volume over time
● Monitor broken links
● Set up 301 redirects where needed
● This report is very helpful after a site re-launch
Social Media Details
● Report Filter: Channel = Social
● Dimensions: Medium, Social Network (or Source)
● ‘Social’ = tagged social campaigns, ‘referral’ = organic social traffic (no camppaign
tags)
Pages by Type
● Report Filter: Page contains <URL identifier for type of content>
● Dimensions: Page
Possible content: blogs, story pages, article pages, FAQ’s
Events by Type
● Report Filter: Event category = ______________
● Dimensions: Event label, event action
Possible events: document downloads, offsite links, navigation links, carousel clicks
‘Unique’ Metrics - Pageviews● To report the number of sessions that
viewed a page, use ‘unique pageviews’
● GOTCHA - do NOT combine page
with sessions as a custom report (GA
WILL let you set this up, but sessions
are ONLY associated with the entry
page)
‘Unique’ Metrics - Events● To report the number of sessions that recorded a certain event, use ‘unique
dimension combinations’
● This shows the sessions with the event for whatever dimension combination is
presented in the report
Custom Segments● Create a custom segment to filter any report by sessions/users meeting specific
criteria
● Some common segments include:
○ Sessions from a specific campaign
○ Sessions that viewed a specific page
○ Sessions that registered a specific ‘event’
● Then apply a segment to a basic or custom report, for example:
○ Geographic reports
○ Traffic (source/medium) reporting
○ Technology: device/browser/OS reporting
Custom Segment - Viewed the homepage●
Custom Channel Groupings● CUSTOM channel groupings give you
the flexibility to roll up the data the way
you want to see it
● They are retroactive, but are specific to
the user account where they are created
but can be shared like other assets
● For the Gazette, we break out
Harvard.edu referrals, other Harvard
referrals, and non-Harvard referrals as
separate ‘Channels’
Custom Channel Groupings● CUSTOM channel groupings are a
view-level setting - be sure to find the
‘custom’ groupings rather than the
‘channel groupings’ (changes to the core
channel groupings will not be retroactive
and will only impact data collection
moving forward)
Helpful Toolkit
Google Tag AssistantChrome Extension
● Quickly check the status of Google
Analytics and Tag Manager code on
any page
● Red/yellow warnings identify tagging
problems
EditThisCookieChrome Extension
● View cookies set on a site
● Delete selected, or all, cookies on the site
without having to clear all of your cookies
for other sites
Google Data StudioGoogle’s new dashboard Tool
now offers free, unlimited
dashboards, with great
integration with Google
Analytics and other Google
products
Features: interactive filters,
flexible formatting, multi-page
dashboards
datastudio.google.com
Takeaways - Top Tip From Each Topic!1. Filters - maintain data integrity by collecting the cleanest data you can in your
production a view (a non-filtered view should also exist), slides 4-12.
2. Settings - key settings to check include referral exclusions (include your own
harvard SUBdomain/s) and make sure bots/spiders are excluded.
3. Reporting - remember to use ‘unique’ metrics when reporting the number of
sessions with a specific page/event.
4. Tools - download ‘Google Tag Assistant’ for very user-friendly feedback on tag
set-up and data collection.
elizabeth@ewbanalytics.com - Feel free to reach out with specific questions!
Copyright © President & Fellows of Harvard College
Responding to Analytics with SEOMarcus Dandurand - March 30th, 2017
How to Measure Traffic from Search EnginesIn Google Analytics: Source/Medium = Google/OrganicIn Adobe Analytics: Marketing Channel = Natural Search
2
Are we getting enough Search traffic?
3
Optimization is never done!
4
➢ Benchmark against yourself
➢ Compare traffic Year-Over-Year
➢ Try optimizing existing pages
➢ Create new pages to target new keywords
➢ Think beyond “branded” keywords
Search Traffic Year-Over-Year
5
Wait… What?!Traffic is down!!! Is it something we did? Did Google change its algorithm?Will it fix itself?What can we do?
6
First, a few SEO myths...
7
1. We don’t know what Google wants
2. The algorithm changes too often
3. SEO is an attempt to “game the system”
4. SEO is a job for IT
5. My CMS has SEO built-in
What do search algorithms care about?
8
Relevance
Performance
Authority
What is page “Relevance”?Your page content closely matches a keyword search phrase.
9
A few ways to improve a page’s relevance:
10
➢ Keyword Research - Relabel content using “outside voice”
➢ Breakup Content - Each important idea should have its own landing page
➢ Accurately describe all page components∙ Page Titles/Meta data∙ Navigation links∙ Section Headers∙ Etc.
What is page “Performance”?Pages are useful, Pages load fast, Site is accessible to humans & robots.
11
A few ways to improve a page’s performance:
12
➢ Reduce page load time
➢ Make sites mobile friendly
➢ Improve click-through rate in search engine results
➢ Make sure websites can be crawled/indexed properly by search engines
What is page “Authority”?Every link acts as an endorsement of a page’s credibility.
Both External and Internal links!
13
Authority began as Google “PageRank”
14
A few ways to improve a page’s authority:
15
➢ Create resources that people will share (inbound linking)
➢ Use 301 redirects (site cleanup)
➢ Restructure website navigation to distribute authority to your most important pages
Distributing Authority:Are the important links on your page 1/10 or 1/100?
Working Knowledge Website Updates:SEO lessons learned the hard way.
16
Mid August 2015
17
Working Knowledge has a search traffic problem!
18
Working Knowledge search traffic improves!
19
Working Knowledge search traffic improves!
20
➢ Sept – Nov 2015: down 43% vs. Prior Year
➢ Sept – Nov 2016: up 46% vs. Prior Year
So, what did we fix?
21
1) New dynamic landing pages
22
➢ Hundreds of Topic landing pages were not indexed by Google. New browse page was behaving like a single dynamic page. (Performance)
➢ Each page had the same Title & Meta Description (Relevance)
2) Deleted the Working Knowledge Archive
23
➢ Several articles were still very popular for search traffic. (Relevance)
➢ Many pages were still cited and linked to by important sources. (Authority)
Without proper redirects, the authority passed back to the home page was lost.
Other optimization for Working Knowledge
24
➢ Created customized Titles & Descriptions for each page in the CMS. (Relevance)
Other optimization for Working Knowledge
25
➢ Created new “display descriptions” visible on-page. (Relevance)
Are You Making A Major Website Update?Please consider the following...
26
#1) Navigation Links Transfer Page Authority
27
➢ Primary navigation create backlinks from every page on your site.
➢ Try to put your important pages in your primary navigation (but only if it makes sense).
➢ Try to remove links that are nice to have, but not critical.
➢ Adding new links will dilute the authority of existing links.
#2) All URL Changes Need 301 Redirects
28
➢ 404 errors create a bad user experience and waste page authority.
➢ 302 redirects are “temporary,” so Google keeps the old page in the index. No authority is passed!
➢ 301 redirects are “permanent,” so the authority of old pages are passed.
#3) Avoid Duplicate Content
29
➢ Each page should have a unique Title & Description.
➢ You should not be able to see the same page via two different URLs.
Pop Quiz:
30
Which of the following URLs below are exactly the same as: www.hbs.edu/mba
A. www.mba.hbs.eduB. hbs.edu/mbaC. http://www.hbs.edu/mbaD. https://www.hbs.edu/mbaE. All of the above
Free Tools for SEO Analytics
31
Free Tools for SEO Analytics
32
➢ MozBar (Browser Extension) ● Page Authority, Inbound links
➢ SiteImprove - Free through HUIT!● Missing Page Titles, Descriptions, Broken Links, 302 redirects
➢ Google Search Console - Formerly “Google Webmaster” ● 404 errors, Organic Keywords, XML sitemap, Mobile issues
➢ Link Redirect Trace (Browser Extension) ● Follow the path of multiple redirects
Thank You!
33
From Big Data to Insights in Massive Open Online CoursesA Traveler’s Guide
Daniel SeatonHarvard UniversitySr. Research ScientistVPAL Research Team
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
• Consortium of Institutions creating MOOCs
• Maintain Open-Source Platform• Host Courses/Content• Lead Outreach• Maintain “https://www.edx.org”
• Partners from Higher Ed / Industry / Government / High Schools
• Create Courses/Content• Manage Courses in Open Online
(MOOC) and On-Campus (residential) settings.
• Perform Research into Teaching and Learning
Consortium Members
What is edX?
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Data from Dec. 2016
http://harvardx.harvard.edu/Harvard University’s MOOC Organization:
• Partners with faculty to create open online courses
• Supports initiatives to use MOOC content beyond open online models
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Data from Dec. 2016
Harvard University’s MOOC Organization:
• Partners with faculty to create open online courses
• Supports initiatives to use MOOC content beyond open online models
http://harvardx.harvard.edu/
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Research Timeline and Perspective
2012
What are learners doing?
Who and where are our learners?
Besides open online, how else can we use MOOC platforms and content?
Why are learners taking courses?
2013 2014 2015 2016
Single MOOC
Transforming Advanced Placement High School Classrooms Through
Teacher-Led MOOC ModelsSeaton, Hansen, Goff, Houck, Sellers
Many MOOCsContext around
MOOC enrollments
Alternative MOOC Models
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Single MOOC
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
• 6.002x: Circuits and Electronics (first MOOC from MITx - now edX)
• Over 100K enrollees• Over 7K certified users• Over 100GB of data from clickstream• Limited profile information
• MITx now a member of edX: ~ 100 open access courses
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
We started with “what” people are doing in 6.002x
Transition between resources
Nodes are resources(size ~ time spent)
Edges are transitions (size ~ weight)
Who does what in a Massive Open Online Course?Seaton, Bergner, Mitros, Chuang, Pritchard ( Comm. of the ACM - 2014)
Analyzed learner interactions with all aspects of 6.002x. Particular focus on time-on-task and resource-use during problem solving.
Measurements• Time-on-Task• Resource Interactions• Daily/Weekly Progress• Transitions between resources
during problem solving
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Many MOOCs +
Context Around Enrollments
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
• HarvardX and MITx Working Paper #1
• Now had access to all course data from MITx and HarvardX
• Addressed “what” people were doing, and “who” they are, across 17 MITx and HarvardX courses
Key Takeaways:
1. Courses are very different.
2. Registrant diversity is immense compared to residential.
3. Participation greatly varies.
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
What are learners doing across MITx and HarvardX?
% G
rade
% Chapters Accessed0
100
100
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
What are learners doing across MITx and HarvardX?
Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Cross course surveys launched in 2014 addressing broad issues across MITx, but teaching experience was central issue.
Results from 11 spring 2014 MITx MOOCs:
• 28.0% (9451) self-identify as past or present teachers (navy).
• 8.7% (2847) current teachers (orange).
• 5.9% (1871) teach/taught the topic (gray).
On average across courses, ~ 8% (1 in 12) of comments are from current teachers.
For teachers that teach/taught the topic, the average across courses is ~6% (1 in 16).
PercentCommentsin Forum
Did not take
survey
Surveyed
SurveyedTeachers
Non-Teachers
43.8%
22.4%
33.8%
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Why are we still not talking about course structure/design?+
Visualizing Course Design
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Goals and Motivations
• Support HarvardX by collecting relevant stats on course structure.
• From a research perspective, identify canonical patterns in course development and better understand how those patterns affect behavior and outcomes.
Human beings, viewed as behaving systems, are quite simple. The apparent complexity of our behavior over time is largely a reflection of the complexity of the environment in which we find ourselves.
- Herbert Simon, “The Science of the Artificial”
Practical Motivation
Abstract Motivation
http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Visualizing Course Design
Key point: Use these visualizations to look across courses.
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Alternative MOOC ModelsAP High School Content/Courses
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
• Boston Public School students• Take MOOC online during school• Commute to BU weekly for
labs/recitations with TAs/Faculty
Of 34 regular and charter schools serving 16,165 students, 2 high schools offer algebra based AP® Physics 1. Only 60 BPS students took the AP® Physics 1 exam during the 2014-2015 school year.
BU Project Accelerate
• Open-online and teacher-led/flipped • All content open on edx.org• Special instances for teachers to
use content in classrooms• Showed 0.08 added to AP exam score per
hour usage above class average
http://vpal.harvard.edu/blog/complementary-models-mooc-instruction-advanced-placement%C2%AE-high-school-courses
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Davidson Next - AP content for use by teachers and students
Program at Davidson College:
• Supplemental content for 14 Challenging Concepts in each AP subject.
• Challenging concepts determined using College Board exam data from 2011 to 2013. Piloted with Charlotte-Mecklenburg School System in 2014-2015 school year.
• Modules designed for each concept meant to facilitate use both in classrooms, and open online. Real AP Teachers from developed content with Davidson faculty.
• Courses released on edX.org and through a new Custom Course tool (CCX).
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Bubble Charts for Detecting Daily Activity• Made these to help monitor teacher use of Davidson Next in Charlotte High
Schools
• Full time assessment coordinator worked with teachers on implementation and efficacy of content (collected district data and AP exam scores).
Transforming Advanced Placement High School Classrooms Through Teacher-Led MOOC Models
Seaton, Hansen, Goff, Houck, Sellers (MIT LINC Conference - May 2016)
Pilot program in North Carolina High Schools
Massive Open Online Courses via edX.org
Exam score residuals are then correlated with student usage relative to class median indicating 0.08 points per hour spent (p<0.05).
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Building Community Around Analytics…
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Open Source Tools for edX Data
• Harvard and MIT already share resources and code for analytics
• https://github.com/mitodl/edx2bigquery
• https://github.com/mitodl/xanalytics
• Open-Source Repos• Python + Google
BigQuery for aggregation of edX data.
• Dashboard via Google App Engine
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
edX Data Workshop Summer 2016
Meeting of data analysts and engineers in institutional roles responsible for edX data; 16 attendees from 11 institutions.
Goals for meeting:
• Discuss broader aspects of data sharing and analytics.
• Standup the Harvard/MIT edX Data Pipeline.
• Happy to report that each participant completed this task.
• Next workshop summer 2017?
• Hoping to broadly release workshop documentation in the spring.
http://news.harvard.edu/gazette/story/2016/07/moocs-ahead/
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Visit our blog: http://vpal.harvard.edu/blog
daniel_seaton@harvard.eduAnalytics Academy - March, 2017
Many collaborators to thank before dicussion!
HarvardXAndrew Ho, Dan Levy, Jim Waldo,
John Hansen, Sergiy Nesterko, Justin Reich, Tommy Mullaney
Miki Goyal, Gabe Mulley,Carlos Rocha, Victor Schnayder,
Olga Stroilova, Brian Wilson
Julie Goff, Aaron Houck, Kristen Eshleman, Pat Sellers, Noelle Smith
Yoav Bergner, Cody Coleman, Isaac Chuang, Curtis Northcutt,David Pritchard, Saif Rayyan
VPAL Research TeamAndrew Ang, Glenn Lopez, Brooke Pulitzer, Yigal Rosen, Dustin Tingley, Selen Turkay, Jacob Whitehill, Joseph Williams
Lydia Snover, Jon Daries, Mark Hansen
Research and
Analytics
Ditch the spreadsheet and tell the story
Katie HammerOffice for Sustainability and Harvard Public Affairs and Communications
How do you get your team to understand your analytics story?
MATERIALS AND CONTENT CREATED:
GHG Landing page (OFS)
4 Page Climate Report PDF
Community-wide message from President Faust
Wide-format Gazette Story and Graphics
8 #HarvardClimateStories instagram profiles
Video targeted at social media
Custom social graphics for 12 Schools + departments
harvard.edu/climate modules
Community-wide email sent by President Faust
52% open rate867 clicks
Gazette Story in the Daily Gazette 22% open rate637 clicks
Social Promotion (Harvard & OFS)
● Video with paid boost● Twitter & Twitter Moment● #HarvardClimateStories
Instagram Campaign
● 108,000+ video views● Almost all Schools shared
news w/ graphics● 17,694 likes; 84 comments
Inclusion in OFS December email newsletter
25% open rate507 clicks
Feature on Harvard.edu 3,010 clicks
External Press (Crimson, Harvard Magazine Story,NY Times, Boston Globe, WGBH etc.)
DISTRIBUTION EFFORTS:
OFS Goal Page: 335
Gazette Story: 210
HUCE Site: 301
Clicks
Try something new?Flaunt it.
FACEBOOK AD CAMPAIGN: OFS
OFS Ad Spend: $170Duration: December 8 - 12
Audience: Targeted students and alumni (where Harvard was listed as School and age was 18+); OFS email list; People who liked our Facebook page
Reach (number of people that saw the post):● 42,019 total people reached● 16,624 people reached as a result of paid● 17,000 total video views; ● Cost per 1,000 people reached $10.23● $.05 per 10 second video view
Engagement (reactions, comments, shares):● 7,696 total actions● $0.27 per engagement ● 23 link clicks● Post generated 55 new GreenHarvard
Facebook page likes
Context:● Typical GreenHarvard video is viewed ~500
times
Note: Many comments did include mention of divestment; however about half were positive and congratulatory.
Qualitative data matters too.
TWITTER STRATEGY:
● Worked with Facility teams to create custom graphics optimized for social for Schools to use
● Partnered w/ HPAC to share on the @Harvard accounts● Outreach in advance to all digital counterparts at Schools/Depts● Created a Twitter Moment to capture various influencer
and School tweets about announcement
RESULTS:
● Initial tweet: Retweeted 66 times; Liked 102 times, Clicked 34 times● Moment tweet: Retweeted 33 times, Liked 92 times, Clicked 90 times● Retweets and original tweets from internal “influencers” like HBS,
HSPH, HAA ● Almost all 12 Schools promoted us in some way, in addition to the
Museums, Libraries, and various departments
And they all livedhappily ever after...
Lessons learned and opportunities
● Targeted outreach to Schools and Depts works; Schools/Depts find easier to promote when can link data/anecdotes back to them (social graphics received well).
● While School/Dept outreach worked and we did have some external influencer tweets (Climate Registry, USGBC), we should develop a more solid plan for faculty and social influencers in the future.
● #HarvardClimateStories campaign a success; 8 profiles in a month was ambitious; for future campaigns could start earlier and conduct interviews and shoots further in advance.
● Important to consider different outreach methods for different audiences; for example the social video was extremely brief but gave an external audience the message they needed “Harvard set an ambitious goal and they met it.”
● We should consider allocating time more evenly across a wide range of projects, considering goals, audiences, and reach (PDFs, videos, social campaigns). For example, though PDF a considerable amount of our time, the social/web reach was minimal.
vs.
Keys to a telling a good story
FormatChoose a vehicle that’s relatable (even if that means powerpoint). Keep it simple.
StyleUse language that seems right for your story (and for your client).
SettingSet your story by bringing in context to explain the why. Remember you control this!
ThemesLet the themes of your data shine by weaving them throughout your story.
IllustrationsImages, examples, and visual cues only add to your story.
ConclusionEnd your story with lessons learned and opportunities that leave the reader ready for your next story!
Thanks!Any questions ?◉ kate_hammer@harvard.edu
Strategize, Synthesize, and JAZZERCIZE®
your Analytics Dashboards and Reports
with your host Aaron David Baker
Remember Jazzercize®?
Problem:
Non-dancers are taking Jazz Dance classes because it is a great workout but aren’t interested in all the work on form and technique. They just wanna have fun while exercising.
Solution:
Create a fun jazz dance-style fitness class that’s interesting and fun!
The power of FUN
The problem was neither exercise nor jazz dance class
Exercise is a chore; we make it fun to get it done.
It’s the same with creating/consuming analytics data
Give them what they want!
The people who consume your reports want them to be engaging.
They don’t always have to contain charts and graphs.
Good document design also conveys professionalism.
Don’t just report on the content you produce or see, share screenshot highlights to give context.
Great reports are persuasive and can change attitudes
Introducing Scoop
HPAC’s analytics dashboard, built around public APIs to Google Analytics, Facebook, Silverpop, and more to come
Presents consistent, up-to-date performance data per story, post, and mailing
Makes comparative metrics possible with benchmarking and visualizations
Introducing Scoop
We have a Strategy
● Identify and capture metrics that matter in one convenient place
We have Synthesis
● Stats from many different platforms in one place for easy reporting
We are working on that Jazz
● Make it beautiful and fun
Strategize
The official Harvard Style Guidelines & Best Practices site has an updated Analytics with resources and setup information.
harvard.edu/guidelines
Use these best practices to ensure your site is up-to-date with the latest analytics code and tracking practices.
First, check that everything is in order
Second, define key stats (these are just some examples)
Health stats are these
● Users/Sessions/Pageviews ● % New Users over time● Page speed● 404s
Strategic metrics are these
● Content performance (pageviews)● Content engagement
○ Time on page○ Scroll depth
● Users by○ New/returning○ Geolocation○ Content sections they visit○ Frequency of visits
● Content dimensions○ Content category/section/tag○ Content length
● Acquisition paths○ Search keywords
Choose goals and metrics that make sense for your organization!
Report on what’s exceptional and on what’s important
What’s exceptional
● Top content in terms of pageviews and/or engagement metrics (time and scroll)
● Large number of people reached or high number of impressions
● Social activity (likes, comments, shares, retweets)
● Unusual spikes in traffic* or unknown sources
What’s important
● Key initiatives○ President Faust’s priorities○ Special Gazette features
● Things you spent money on○ Paid social, AdWords, etc.○ Extra money gives you extra
metrics● Experiments
○ A/B testing ○ SEO
● Conversions
Here we’ve chosen to highlight pageviews by channel and accumulated pageviews over time
● Helps us visualize content distribution and sources of traffic.
● Benchmarking is our own comparative metric—average daily pageviews of all stories in Scoop.
Scoop Example
Synthesize
The chore of reporting
● Go here● Find stats● Copy/paste into doc● Go there● Find stats● Copy/paste into doc● …
Wouldn’t it be nice if the stats gathered themselves?
Scoop pulls in stats from
● Google Analytics● Facebook● Silverpop
With plans to incorporate
● Twitter● Instagram● YouTube● Etc.
Anything with an API can be consumed.
Scoop Example
Synthesis isn’t just about automation
Automation is nice and does save a lot of time.
Linking things by a common element (like URL or topic) can make finding the stats easier.
Synthesis is really about telling the whole story.
Automated reporting cannot speak for you.
Talk about the why in your reports.
My Weekly Report
Ultimate synthesis of what happened last week
Mostly highlighting what’s exceptional
Occasionally mentioning what is bizarre
Always as interactive as possible
● links go to actual online posts or to Scoop itself
Jazzercise!
Back to Jazzercize®
Our reports are working, and people come to us for information, but how can we analytics reports more enjoyable?
Design and data visualizations:
● Beautiful, clean, contemporary, and inviting design
● Display visually our data’s trends, patterns, and correlations
● Provide content creators and distributors with intuitive, at-a-glance insights about performance of published work
● Be designed with interactive development in mind
The plan for Scoop
Your turn to talk:
How do you add jazz to your reports?
Thanks y’all!