Date post: | 19-Aug-2015 |
Category: |
Data & Analytics |
Upload: | ioannis-yanni-antonellis |
View: | 168 times |
Download: | 2 times |
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Machine Learning at Upwork
Yanni AntonellisSenior Director, Data [email protected]://www.linkedin.com/in/ioannisantonellis
ICML 2015 Workshop on Crowdsourcing and Machine Learning
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Outline
• Machine Learning at Upwork
• Upwork Freelancer Search
• Upwork Job Search (Alya Abbott)
2
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Competitive Time-consuming
Expensive Inflexible
“Traditional” work models, remnants of the Industrial Age, don’t make sense today
4
There are 53 Million freelancers in the U.S. today
Source: Staffing Industry Analysts
By 2020, nearly half of the U.S. workforce will be freelancing
$47B online staffing
$45B
2013 2020
Source: Freelancers Union
$47B
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Commerce has moved online and now services, including work, are too
6
workretail lodgingtransportation
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Online Platforms Enabling Steady Growth in Product Commerce & Enabling Rapid Growth in Service Commerce
7
Internet Trends Report by KPCB & Mary Meeker
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
The Internet is becoming the road into work
8
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Future teams will look like movie crews
9
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Companies that aren’t distributed can’t possibly
say they hire the best people…the world is a big place.Tom Preston-Werner, Founder of GitHub
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute. 11
4M+businesses
seeking 2,500+ skills
10M+freelancers
from 180+ countries
$1B+worth of work
annually
Upwork today
Our marketplace connects clients with talent in a snap
2.4MJobs
posted annually
22Candidates
per job
24 hrs25% of jobs
are filled
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Clients of all sorts hire and manage online
Startups Mid-sized Large
5+ freelancers
30+ freelancers
300+ freelancers
13
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Freelancers find clients and work online
14
Small agencies & independent professionals
Moonlighters Studentsand retirees
Anna · Designer
UK
Stan · Data Scientist
Florida
Esteban · DeveloperArgentina
Data
We have the biggest closed-loop online dataset of jobs and job seekers in labor history
Resumes (~10M)Job Posts (~10M)Job Applications (~100M)Messages (~100M)Hiring decisions (~10M)
Contract progress (~1B)Feedback(~10M)Web site activity (~10B)Money transactions (~100M)
Match: How clients find freelancers
1. Freelancer search
2. Similar freelancers
3. Freelancerrecommendation
s
Match: How freelancers find interesting jobs
1. Job feed/recommendations
2. Job search
3. Similar jobs
Match: How clients pick the best applicants
1. Sort by Best Match
2. Filter spam
3. Guaranteed applicants
Match: What makes it hard
Personalization
Two-sided marketconstraints
“gangnam style video” “iOS developer”
Each contractor has ~170 work hours
per monthin 7 months
1. POST – APPLYContactors apply to posted jobs, clients evaluate the applicants and decide which one to hire
2. SEARCH – INVITEClients search for contractors and invite them to work on a posted job
Two ways for contractors and clients to find each other...
1. POST – APPLYContactors apply to posted jobs, clients evaluate the applicants and decide which one to hire
2. SEARCH – INVITEClients search for contractors and invite them to work on a posted job
Two ways for contractors and clients to find each other...
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Client Funnel & Search/Recommendations
SignupJob Posting
InvitationsAccepted Invitations
HiresGood Outcomes
$
26
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Client Funnel & Search/Recommendations
Visitor site search
User site Search & Recommendations
SignupJob Posting
InvitationsAccepted Invitations
HiresGood Outcomes
$
27
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Client Funnel & Search/Recommendations
Visitor site search
User site Search & Recommendations
Search before a
job posting
Search after a
job posting
Freelancer recs after a job posting
Freelancer recs after
an invitation
Applicant Ranking
SignupJob Posting
InvitationsAccepted Invitations
HiresGood Outcomes
$
28
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Client Funnel & Search/Recommendations
Visitor site search
User site Search & Recommendations
Search before a
job posting
Search after a
job posting
Freelancer recs after a job posting
Freelancer recs after
an invitation
Applicant Ranking
SignupJob Posting
InvitationsAccepted Invitations
HiresGood Outcomes
$
29
Ranking
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Client Funnel and Search
• Different flavors of Ranking are needed per Search Product since:
– (a) the objective we are optimizing for and
– (b) the available features vary.
• For example, when a client searches for freelancers before he has posted any job we probably want to optimize the results for encouraging the client to post a job or contact a freelancer.
• However, when a client is searching *after* posting a job, we have the job context information and we probably want to optimize the search results for accepted invitations or hires.
30
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Optimization Objectives
Data Product Optimization Objective
Visitor search Signups, FJPs
Search before a job posting Job postings, Invitations
Search after a job posting Accepted Invitations, Hires, Good Outcomes
Freelancer recommendations after a job posting
Accepted Invitations, Hires, Good Outcomes
Freelancer recommendations after an invitation
Accepted Invitations, Hires, Good Outcomes
Applicant Ranking Hires, Good Outcomes
SignupJob Posting
InvitationsAccepted Invitations
HiresGood Outcomes
$
31
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Optimization Objectives
Data Product Optimization Objective
Visitor search Signups, FJPs
Search before a job posting Job postings, Invitations
Search after a job posting Accepted Invitations, Hires, Good Outcomes
Freelancer recommendations after a job posting
Accepted Invitations, Hires
Freelancer recommendations after an invitation
Accepted Invitations, Hires
Applicant Ranking Hires, Good Outcomes
SignupJob Posting
InvitationsAccepted Invitations
HiresGood Outcomes
$
32
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Search after posting a job - Funnel
33
Impressions Clicks InvitationsAccepted Invitations
HiresGood Outcomes
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Search after posting a job
Impressions Clicks InvitationsAccepted Invitations
HiresGood Outcomes
Relevance Availability Eagerness Quality
34
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Relevance
• Freelancer summary
• Job summary
• Freelancer - Job fit
• Freelancer - Client fit
35
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Availability
Sample Features
● Recent hours worked
● Recent invitations received
● Recent applications placed
● Recent site location use:
o Contractor profile
o Job search
o Message center
● Contractor-provided availability
● Historical wages & current quote
36
Impressions Clicks InvitationsAccepted Invitations
HiresGood Outcomes
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Eagerness
AUC of 0.792 - Sample features:
● Job applications involving the freelancer:
o Job apps sent/invites received in previous {1, 2, 4, 7, 14, 30} days
o Job apps/invites filled in previous {1, 2, 4, 7, 14, 30} days
o Job apps/invites rejected in previous {1, …, 30} days
o Job apps/invites cancelled in previous {1, …, 30} days
● # of Eventlog entries at {message_center, job_search,
contractor_profile_page} locations in previous {1, …, 30} days
● Hours worked in previous {1, …, 30} days
● Profile details:
o Num fixed/hourly jobs worked
o Typical wage; rate quote
o Self-reported availability
o Num. active assignments and candidacies
o Num. jobs worked in each category group
37
Impressions Clicks InvitationsAccepted Invitations
HiresGood Outcomes
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Quality
AUC of 0.61 - Sample features:
● Average number of Good outcomes in the past● Average number Bad outcomes in the past● History of Freelancers in terms of receiving/giving bad or good private
feedbacks● Profile exams taken (e.g. PHP test, Book keeping test, Spanish Spelling Skills
test )● Freelancers Skills (e.g. Editing, Translation, Proofreading, . . . )
38
Impressions Clicks InvitationsAccepted Invitations
HiresGood Outcomes
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Search after posting a job
Impressions Clicks InvitationsAccepted Invitations
HiresGood Outcomes
Relevance Availability Eagerness Quality
Visibility Rotation
39
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
Summary
• Many flavors of freelancer search based on:
– optimization objective, context
– data/features available
• Key building blocks
– Relevance
– Availability
– Eagerness
– Quality
– Visibility Rotation42
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute. 43
Hiring in 3 minutes is … magic
© 2015 Upwork Inc. Proprietary and confidential. Do not distribute.
ThanksYanni Antonellis, Senior Director, Data Science