Date post: | 24-Jan-2018 |
Category: |
Technology |
Upload: | papisio |
View: | 434 times |
Download: | 0 times |
Solving a business problem in just two weeksusing machine learning
Lucas F Navarro and Pedro [email protected]@getninjas.com.br
PAPIs Connect - São Paulo 2017
GetNinjas’ Data Science Squad
Product OwnerBs’ in Computer Science at UFSCarMs’ in Artificial Intelligence at UFSCar
Data ScientistMechatronics Engineer from
USP/CAASO
Data ScientistBs’ in Applied Maths at UNICAMPPhD’ in Artificial Intelligence at USP/POLI
Python DeveloperBs’ in Information System at USP/EACH
BackEnd DeveloperComputer Engineer from
USP/POLI
Agenda
1. How does GetNinjas Work
2. The current Business Model and why we chose it
3. The big problem that has emerged
4. How we solved this problem in just two weeks
5. Results we achieved
6. Touchpoints
How does GetNinjas work?(Client)
1. Choose what kind of service that you need.
2. We indicate up to 4 pros at your region
3. Hire a Pro to solve your problem
How does GetNinjas work?(Pro)
1. Choose a service request from the store
2. Use credits to unlock the contact of the client
3. Get in touch with the client to be hired
OVERVIEW
2,000,000services hired in the last year
200,000professionals registered
R$ 210M+in transactions concluded (GMV) over the last 12 months
100+categories ranging from electrician to english teacher to lawyer and so on...
We were performing a staged rollout of a new pay-per-lead model, with significant improvements to the pros' experience
Old ModelSubscription
- Pro Pays Fixed Value Per Month- Receives up to
30 contacts
New ModelBundle
- Pro Purchases a Bundle of credits- Use credits to
purchase contacts
Vs
First Launch: South (Sep/18)
Second Launch: North + Center (Oct/23)
At last:SP & RJ (2017)
Main improvements:
● Pros can purchase how many contacts they want, when they want
● Clients will always receive interested Pros
● More feasible to balance Supply & Demand
Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered
Next launch was in two weeks!!!
29%of the requests were not
being sold
We would have 43.500 clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure out what causes this rate
Some of the possible reasons:
○ Price○ Location○ Clients only estimating price○ Requests lacking a good description○ Miscategorized services○ Not enough pros with credits
How to solve all of this in two weeks?Our next launch was much bigger and we could not leave our clients unanswered.
Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered
Next launch was in two weeks!!!
29%of the requests were not
being sold
We would have 43.500 clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure out what causes this rate
Some of the possible reasons:
○ Price○ Location○ Clients only estimating price○ Requests lacking a good description○ Miscategorized services○ Not enough pros with credits
How to solve all of this in two weeks?Our next launch was much bigger and we could not leave our clients unanswered.
Despite offering higher flexibility to pros, the pay-per-lead model can leave some clients unanswered
Next launch was in two weeks!!!
29%of the requests were not
being sold
We would have 43.500 clients/month without response
(Around 0.5M per year)
We made quantitative and qualitative analyses to figure out what causes this rate
Some of the possible reasons:
○ Price○ Location○ Clients only estimating price○ Requests lacking a good description○ Miscategorized services○ Not enough pros with credits
How to solve all of this in two weeks?Our next launch was much bigger and we could not leave our clients unanswered.
The problem gets bigger!
Each point is a “Mini GetNinjas”, with 100 categories
300 cities
100 categoriesx
30k pairswhich we have to deal with that problem
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have Pros available to solve your problem at this moment”
2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url”
11%of the Free Pros were purchasing a bundle after seeing the services store at our App
Only 4% of our subscribed pros were payees at that time
We had the potential to more than double our payees base if we sustained that conversion rate
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have Pros available to solve your problem at this moment”
2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url”
11%of the Free Pros were purchasing a bundle after seeing the services store at our App
Only 4% of our subscribed pros were payees at that time
We had the potential to more than double our payees base if we sustained that conversion rate
What if we could predict that a client request wouldn’t be purchased?
1st option, tell the client the truth
“Unfortunately we don’t have Pros available to solve your problem at this moment”
2nd option, let’s give this services for free to signed up pros that never purchased a plan! ✔
“GetNinjas: Congrats, you won an electrician service request near your location! Get the contact in “My Requests” tab at our app: %url”
11%of the Free Pros were purchasing a bundle after seeing the services store at our App
Only 4% of our subscribed pros were payees at that time
We had the potential to more than double our payees base if we sustained that conversion rate
How could we predict that a client request wouldn’t be purchased?
We had already figured out the reasons that a service was not sold, so we used it as features and started to test some machine learning models using R.
Main R packages used:
For data prep:○ tidyr○ dplyr
ML algorithms:○ monmlp (neural net)○ e1071 (naivebayes)○ party (decision trees)
Utilities:○ ROCR (ROC curve)○ SDMTools (confusion matrix)○ Boruta (feature selection)○ Pool + RPostgreSQL (integration
for our data warehouse - RedShift)
Some of the features:
● Number of Pros with credits*● Avg Distance from pros*● Number of highly reviewed pros*● Price● Does service has a text description?
Class: Was the request sold? (boolean)
We used Rstudio Server remotely from a m4.large EC2 machine at Amazon’s Cloud
*Just pros inside the request area (inside a radius)
From R to Python, Fujin was Born!
We came up with a Neural Network with great performance:
○ With this specificity rate, just 30% of the unsold requests clients would remain without an answer
○ With this sensitivity rate, we would distribute only 10% of the good requests for free
After we validated the model, we implemented it in python to go live:
○ We used the basic DS set (scikit-learn, pandas, numpy)○ Alse we used Neurolab for the MLP algorithm○ We used Flask to create the prediction microservice
70%of specificity
First Model
79%of accuracy
*Python is the main Data Science language at GetNinjas
90%of sensitivity
We called the project Fujin!
How fast this entire project was delivered?
To have a functional ML model to predict unpurchased requests we spent just around 2 weeks!
*The first distribution algorithm was a random distribution to Free Pros
In Its first month live, Fujin were able to predict 68% of the unpurchased requests
Unpurchased
Predicted/Distributed
Contacted
Hired
41%
28%
9%
5%
Service
RequestsWith north+center regions, unpurchased request rate grown from 29% to 41% due sparsity of S&D in this new regions
With Fujin online, the number of unanswered clients were just 13% (instead of 41%)
Fujin also helps to increase our Gross Merchandise Volume(GMV) through hiring rate
*Numbers from our requests of November/2016
Fujin was also helping to activate Free Pros into Payees
*Numbers from our requests of November/2016
≈R$25k in purchases from Pros after they received a free client contact from Fujin
65% of this pros didn’t even had our App installed
⚠ Only 14% of the distributed leads were read by the Pros
Already had our App installed Downloaded App post Fujin lead Didn`t Download
Towards to the Future
● The distribution algorithm○ Pick the best pros to send free requests○ Determine the amount of free pros to send the contact from a request given the Fujin’s
feedback
● Enhance the model with more features○ We have more service categories with distinct characteristics nowadays○ We have SP and RJ state (higher density regions than before)
● Features to increase the visualization rate of free distributed requests○ Such as adding a deeplink to our the contact screen on our App at the e-mail/sms sent to Pros
Touchpoints
● The non-bureaucratic/non-hierarchical organization of a StartUp company was crucial for us to develop Fujin in such short time
● ML is amazing, in just two weeks we had a running model to solve the biggest part of our problem
● ML is super cool, but most of the times we solve our problems with much simpler approaches
⚠ Do not create a super complicated solution and later adjust your problem to it, start from the problem, and climb to the complex solution as you need it.
Thanks for your attention!!!
Questions?