Polly

Doctoral Thesis Proposal

Viral Entertainment as a Vehicle for Disseminating Core Development ServicesAgha Ali Raza

Thesis Committee:Roni Rosenfeld, LTI, CMU (Chair)Jaime Carbonell, LTI, CMUBhiksha Raj, LTI, CMUUmar Saif, ITU, PakistanBill Thies, MSRI, IndiaLanguage Technologies Institute1ContributorsCMUChristina MiloJehanzeb SherwaniGuy AlsterYibin LinHaohan WangRita SinghChristos FaloutsosDanai KoutraJay YoonLUMS/ITU (Polly-Lahore)Umar SaifMansoor PervaizSamia RazaqFarhan Ul HaqZain TariqBabajob.com (Polly-Babajob)Sean BlagsvedtMaya ChandrasekaranArchna Bhatia

MSRI (Polly-MSRI)Bill ThiesIndrani MedhiSpandana Gella

IIT Delhi (Polly-JMV)Zahir KoradiaAaditeshwar Seth

IntroViral Ent.ConduitTest-bedReproducible2ICT4DInformation and Communication Technologies for Development

More and better information and communication furthers the development of a society.

Can lead to better management of available resources, improved monitoring & reporting of corruption and more connectivity among people

To achieve impact at a massive scale:Robust solutions to reach the masses using available means with minimum resource expendituresIntroViral Ent.ConduitTest-bedReproducible3Can facilitate development where information and connectivity are the missing components

How do you reach low-literate masses?TV, radio are non-interactivePCs are not feasibleSmart phones are not always feasibleText is problematic for the non-literate

Speech over simple phones is a viable way to reachlow-literate masses

IntroViral Ent.ConduitTest-bedReproducible4PCs are not feasible for low-literate massesNon-affluent cannot afford themUnskilled cannot operate themStable electricity, Internet etc. cannot be relied uponLow cost phones may be a feasible solution

Low-cost phones often only support voice & SMSVideo not feasibleGraphical interactions not always possible

Text is problematic for low literate usersSpeech may be the only viable interaction paradigm

Long Term VisionSpeech-based information access for low-literate people in developing countries

Examples of information services:speech-based Craig's Listspeech-based citizen journalismspeech-based message boards/blogshealth, agriculture, jobs, education,IntroViral Ent.ConduitTest-bedReproducible5We want to facilitate development among low-literate usersWe want to do that by providing them access to information and communication technologiesAnd we believe that it can be effectively done using speech based services over simple phones

We envision information services like Speech based access to:health informationagricultural informationJob search

And communication services like, speech-based:Market placeMessage boards and blogsCitizen journalismProblem 1: User Training & MotivationSpeech Interfacesusually require user training e.g. Healthline (Sherwani et al 2009), Avaaj Otalo (Patel et al 2010)

Explicit training is not a scalable solution

Solution: Incentivize people to train themselvesWhere there's a will there's a way [Smyth et al 2010]

Entertainment turns UI Barriers into mere speed bumps

IntroViral Ent.ConduitTest-bedReproducible6Even simple IVR systems are often a challenge for the low-literate

6Problem 2: Mass DisseminationHow to advertise / communicate to masses who dont read?

Solution: Viral Spread

IntroViral Ent.ConduitTest-bedReproducible77GoalsDevelop Viral Entertainment as a vehicle for disseminating Development related telephone based services

Research Objectives

Find the right kind of EntertainmentUse it to Introduce and Popularize speech interfaces Use entertainment as a Viral Conduit for delivering Core Development ServicesSetup an Experimental Test-bed for testing speech interface choicesIntroViral Ent.ConduitTest-bedReproducible8Can we find a:simple-to-understand, non-controversial, quick-to-engage-and-spread form of entertainment that is suitable for low-literate telephone users formerly inexperienced with automated dialog systems?

Could it become Viral?

Research QuestionsIntroViral Ent.ConduitTest-bedReproducible9A telephone-based, voice-based application which allowed users to listen to songs recorded by others, as well as to record their own songs and to forward them to friends

Our First Attempt: Songline

User Feedback (Pakistan)About the idea:Songs and music are considered controversialPrivacy concerns

About the Interface:Entering phone numbers is tediousBusy tone is confusingDetailed call tree and numerous options are confusingIntroViral Ent.ConduitTest-bedReproducible1010Polly

Polly is a telephone-based, voice-based application which allows users to make a short recording of their voice, modify it and send the modified version to friends.

FeaturesNon-controversial entertainmentPeer-to-peerSimple & Easy-to-understand (few options)

IntroViral Ent.ConduitTest-bedReproducible11112011-Pilot LaunchSeeded with the 32 users on March 31, 2011Remained up for 22 days Amassed 2,032 users who took part in 10,629 interactions

IntroViral Ent.ConduitTest-bedReproducible12122011-Pilot LaunchSeeded with the 32 users on March 31, 2011Remained up for 22 days Amassed 2,032 users who took part in 10,629 interactions

Reasons for Shut down:Line capacity (single phone line) saturated within two weeksLine remained busy during peak hours resulting in frustrated users The international call charges ($0.126/minute) were becoming a significant financial burden for us

People kept calling Polly 40 days after its shut down when we finally stopped monitoring the calls.

IntroViral Ent.ConduitTest-bedReproducible1313Can we use Entertainment as a delivery vehicle for core development services?

Research QuestionsIntroViral Ent.ConduitTest-bedReproducible14

2012 Large Scale DeploymentDeployed locally in LahoreCall charges went down to $0.023/minuteIncreased capacity to 30 phone linesIncreased number of voice manipulations to sixAdded Job Audio-Browser

IntroViral Ent.ConduitTest-bedReproducible1515First Information Service: Job Audio-BrowserScan Pakistani newspapers for jobs for low-skilled workers

Record these adsInvite Pollys users to audio-browse them

Ref: paperpk.comIntroViral Ent.ConduitTest-bedReproducible1616IntroViral Ent.ConduitTest-bedReproducible

17172012 Large Scale DeploymentLaunched on May 09, 2012:Cold seeded with 5 most-frequent users of the pilot

After 141 days (mid-September 2012):495,000 calls85,000 usersSpreading to 1,000 new people daily

27,000 people used the job search servicelistened 279,000 times to job ads and forwarded them 22,000 times to their friends.

IntroViral Ent.ConduitTest-bedReproducible18

2012 Large Scale Deployment

IntroViral Ent.ConduitTest-bedReproducible192012 Large Scale Deployment

IntroViral Ent.ConduitTest-bedReproducible20After a YearPolly had accumulated:636,000 calls165,000 users

34,000 people used the job search servicelistened 386,199 times to 728 job ads and 19,000 users forwarded them 34,000 times to their friends.

IntroViral Ent.ConduitTest-bedReproducible21Can we use the virally spreading entertainment service as an Experimental test-bed for performing randomized controlled trials and demographic studies?Research QuestionIntroViral Ent.ConduitTest-bedReproducible22

User Retention

A few people continue long term

Enthusiasm is Lost Quickly

IntroViral Ent.ConduitTest-bedReproducible2323

Randomized Controlled TrialsSo far we had been paying all airtime charges using missed call mechanismGoal: Reduce our airtime charges while maintaining system spread among the poorQuestion: How will various quotas affect user behavior?

Introduced a caller-paid lineCalls picked up, caller pays airtimeImpose various quotas on the number of toll-free calls per day (for each user). When quota exceeded, direct caller to caller-paid lineIntroViral Ent.ConduitTest-bedReproducible24

Caller-paid Line

Significantly less activity as compared to the toll-free lineIntroViral Ent.ConduitTest-bedReproducible25

RCT: Effect of Daily Quota of 7 Calls

No appreciable difference after a weekIntroViral Ent.ConduitTest-bedReproducible26User Demographics

Determined by listening to a sample of recordings:

Used mostly by Punjabi speaking men

IntroViral Ent.ConduitTest-bedReproducible27undetermined (young children, old people, too much background noise, etc.).5388 recordings27User DemographicsEstimated from 207 survey calls

Low SES(No Education)Socio-Economic StatusPrimarily used by low-educated, low-income peopleIntroViral Ent.ConduitTest-bedReproducible2828Geographical Spread

- Seeded in Lahore and Okara

- Reached all parts of Pakistan.

- And also a handful of calls from:IndiaBelgiumOmanSaudi ArabiaUAE

IntroViral Ent.ConduitTest-bedReproducible29Analysis of User BehaviorDoes more experience using Polly lead to:

an improvement in users interaction skills?Use of more advanced features,Fewer mistakes.

2. any change in usage preferences?

(Analyses are based on: 50,414 users, 292,951 calls, 934,742 menu interactions)

IntroViral Ent.ConduitTest-bedReproducible30

Barge-in Behavior in Main Menu

Clear increase in the use of barge-in with experienceIntroViral Ent.ConduitTest-bedReproducible31

Number of times user previously encountered main menuFailed Forwarding attempts

Clear decrease with experienceIntroViral Ent.ConduitTest-bedReproducible32

Early Differences in Call ComplexityCan be used to predict user retention!

IntroViral Ent.ConduitTest-bedReproducible33

33Reproducibility: Can our setup and results be reproduced in a different country?

Research QuestionIntroViral Ent.ConduitTest-bedReproducible34

Babajob.com is one of Indias largest informal and entry level job portal that connects registered job seekers and employers using their website (primarily), voice services as well as SMS(Another benefit: A limitation of our previous study was our inability to track employment outcomes)

Goal: Advertise Babajob to low-literate job seekers through Polly and connect them to Babajobs voice services

The Babajob Collaboration(Sean Blagsvedt, Maya Chandrasekaran)IntroViral Ent.ConduitTest-bedReproducible35

A local phone number in Bangalore to receive the missed callsPolly calls back from the USSupport for Kannada (Archna) and Hindi (Rita) voice promptsCall transfer to Babajob IVR system from Pollys main menu.We added a feature: directory of previously-called friendsSetupIntroViral Ent.ConduitTest-bedReproducible36

Automated calls to 84 active Babajob users in the Driver category followed by an SMSStaff members at Babajob volunteered to pass Pollys number on to 5 of their friends

ResultsVery little activity that declined within a few weeksVery few users; mostly listening to job ads (not much playing or forwarding)We suspect these users to be Job Data brokersSeeding Attempts (via Babajob)IntroViral Ent.ConduitTest-bedReproducible37

On Jan 9, 2014 a set of undergraduate students were asked to spread word about Polly. They posted on their university Facebook pages and message boardsIn the last week of Jan, 2014 Spandana gave Pollys demo to a security guard in her building and asked him to spread the word

ResultsOver the last 106 days:5,051 successful calls 2,793 call-back requests (from 441 users)2,258 message delivery calls (to 699 users)

Seeding Attempts (via MSRI)(Bill Thies, Indrani Medhi, Spandana Gella)IntroViral Ent.ConduitTest-bedReproducible38

MSRI Seeding: Viral, but Not ExponentialIntroViral Ent.ConduitTest-bedReproducible39

Neither exponential growing nor decaying!Steady stream of new usersBeing used primarily by low-literate users in West Bengal and Delhi

Still OngoingWhy isnt Polly Spreading Exponentially in India?IntroViral Ent.ConduitTest-bedReproducible40Back-Channel SpreadObservation: Polly is often introduced to new users not via a Polly messageIndication of this: some users place their first call without ever receiving a Polly message (Word of Mouth users)47.5% of call-initiators in Polly-LHR pilot15% of call-initiators in Polly-LHR full deployment18% of call-initiators in Polly-BLR

In person introduction? +demonstration?Explained in a phone conversation?

IntroViral Ent.ConduitTest-bedReproducible41

Why is Spread Non-Exponential? (Hypotheses)Introduced as a free voice messaging systemDifferent perception perpetuated by back-channel spread?Introduced to college undergraduatesDifferent behavior in this SES?Indians have bad past experience with IVR systems?SPAM, scamsComplex billing mechanisms due to telecom circles (=area codes)?Linguistic complexity?Dead-Skunk hypothesis: something subtle is broken in the user experienceCultural differences: Users expressed less interest in voice mods

IntroViral Ent.ConduitTest-bedReproducible42

JMV-collaboration (Zahir Koradia, Aaditeshwar Seth)Jharkhand Mobile Vaani is a citizen radio-over-phone platform that makes recorded radio content accessible through phone callsGoal: Cross-spread: Polly and JMV

Benefits:JMV operates in the Hindi-BeltThey already have a trusting user-baseMore flexibility of seedingThey have a vested interest in this venture

Questions: Can Polly grow exponentially in India?Will Polly-JMV cross-spread?

Remaining Work43

Launch Polly-JMV (May 2014)Complete analysis of Polly-BLR data and remaining experiments (May, June 2014)Analyze data from Polly-JMV (June, July 2014)Write it up! (July 2014)Thesis Defence (end of July, 2014)

Thank you!Questions?

Remaining Work44

LimitationsI am not on-groundI do not understand the culture/speak the language

Either some inherent cultural/sociological difference that we do not appreciateWrong system image (a voice SMS system) due to incorrect seeding? (Back-channel hypothesis).Peoples' past experience with IVR systems has left a bad taste? In Pakistan it was an untouched population.Phobia of pressing keys lest thy may lose money?Fear of calling/picking up calls from another telecom circle?More entertainment options? More free messaging options? Is the entertainment void situation among low-SES Indians similar to their Pakistani counterparts?Busy people, less time.

Hypotheses regarding Non-Virality45

Some system bug that we have not been able to isolateAre we getting all the missed call requests from all the regions?Is the SMS okay. (No!)Is the caller ID okay? (it was not. Just fixed.)Are there pauses? Delays? buggy IVR? (None that we are aware of)Is the language okay? (we believe so)

Hypotheses regarding Non-Virality46Microphone vs telephone based prompt recordingsTranslation of prompts. Formal vs informal.Outgoing caller-id problem.Quick detection of bugs. The silent voice prompt.Problems in seeding.On ground presence and understanding the local culture, norms, language.BLR Lessons LearnedCaveat: Confounding User TypesThis analysis lumps together long-term and short-term users.Hence, it confounds true learning by any one user with differences between the different user types.Long-term users may be more adept at using IVR systems to start with. Changes in User Preferences:(now controlling for user type)We define 3 user sets and explore changes in their usage patterns as a function of their experience (number of prior Polly calls):

Short-term users: 2,701 users who interacted with Polly exactly 5 times.Intermediate-term users: 1,862 users who interacted with Polly exactly 10 or 11 times.Long-term users: 1,523 users who interacted with Polly 30+ times.Menu Choices vs. ExperienceShort-term Users (made exactly 5 calls each)

Change Y axis label to Avg. # responses per call, per user.Remove spurious X axis tick marks (0.5, 1.5, etc.)

50Menu Choices vs. ExperienceIntermediate-term Users (made exactly 10 or 11 calls)

Compared to short-term users; activity starts at a higher level and climbs higher!Menu Choices vs. ExperienceLong-term Users (made 30+ calls)

Compared to intermediate-term users; activity starts at a higher level and climbs higher!Common Trends among all user sets- The Tendency to:1. Press 2 (forward) starts off at a high value and stabilizesBack ChannelSystems popular image as a messaging system 2. Press 3 (next effect) increases with experienceSystem ExplorationHunting for an effect of choice 3. Press 0 (re-record) and 1 (repeat) decreases with experienceAdapting to 15 seconds recording limitPlaying with the system, alone or with friends

- Invalid button presses are rare

ContributionsMulti-platform support (Voxeo Tropo, FreeSwitch)

Ability to do quick remote pilot launch followed by local deployment

Much more analysis to be doneDetailed logs, audio recordings,...

Future WorkCross sell Polly-Jharkhand Mobile Vaani

Re-launch Polly in Pakistan to serve as a clearinghouse for jobs and skill-training opportunities

Make Polly available over VBTS (Village Base Station)

Polly-Pilot: Main Use

Number of recordingsIntroGoalsPollyAnalysisPlansRemaining percentage goes to unclear recordings Histograms of numbers

56Polly was used forAmong a sample of 63 users (reached through survey calls):34 used it for fun; 17 for utility; and 12 for bothFun:making prank calls to friendshello-hi/random messages, Poetrybrowsing job ads as a pastime! Utility: More serious users defined Polly as a voice messaging system that they use to:send occasion (holiday, birthday) greetings, to request a call-back, to know a friends whereabouts,to browse and apply for jobs.

Four blind users defined Polly as an alternative to text messaging and praised it profusely.

Demographics and what Polly was used forCountEntertainment*Utility*All recordings39974%56%By GenderMale35973%58%Female2673%38%Unclear14By LanguageUrdu6456%53%Punjabi19468%42%English4100%0%Pushto (est.)10093%77%other/mixed37* Categories are neither comprehensive nor mutually exclusive.Demographics and Usage patterns of caller-paid callersCountEntertainment*Utility*All recordings538825%75%By GenderMale471326%79%Female59023%81%Unclear93By LanguageUrdu113513%89%Punjabi348029%76%English2341%59%Pushto (est.)70330%76%other/mixed45* Categories are neither comprehensive nor mutually exclusive.Demographics and Usage patterns of toll-free callers