31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
1/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
ESCO Mapping Pilotworkshop 4
03/11/2015Brussels
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
2/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Agenda
Time Agenda item Time
10:00 Opening of the meeting 12:45 General conclusions mapping classifications
10:20 Status update mapping pilot
13:45 Coffee break
11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching
11:15 Country specific conclusions mapping classifications
14:45 Next steps & planning
12:00 Lunch 15:00 End of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
3/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Opening of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
4/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Approval of meeting minutes July 2nd
• Meeting minutes of the July 2nd workshop• Feedback of Julius de Zeeuw on section Dutch data
• Feedback of Alain Dupuch on section French data
• Feedback of María José Arias Fernández on section Spain data
• Other corrections
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
5/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Agenda
Time Agenda item Time
10:00 Opening of the meeting 12:45 General conclusions mapping classifications
10:20 Status update mapping pilot
13:45 Coffee break
11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching
11:15 Country specific conclusions mapping classifications
14:45 Next steps & planning
12:00 Lunch 15:00 End of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
6/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Status update mapping pilot
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
7/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Mapping pilot objectives
• Define a straightforward process to create a mapping
• To prototype tooling to support the mapping process
• To create draft mappings for NL, FR, CZ, and ES (Hospitality and Tourism)
• To validate the draft mappings to learn
• What type of information is retained and what is lost
• What is going well, and what needs improvement
• To test the mappings with some CVs and Job Vacancies
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
8/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Mapping process
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
9/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Actions from previous workshopTask Who
1 Provide an updated mapping tool• User validation interface• Enhanced ROME occupation proposals based on appélations• Enhanced skill mapping proposals based on skill/occupation
relations
TenForce
2 Participants provide feedback on mappings PESs
3 Select & translate curriculum vitae and job vacancies for matching TenForce
4 Process curriculum vitae & job vacancies PESs
5 Process cross-border curriculum vitae and job vacancy matching TenForce
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
10/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Updated mapping tool –strategies to improve proposed mapping
1. SolR based text matching[new things below this line]
2. Automatically add synonyms
3. Better filtering using ISCO-08
4. Match concepts based on context
5. Use hierarchy
6. Use skill relations
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
11/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Updated mapping tool- subject based clustering
• Using open source Mallet NLP engine (UMASS)
• Use extracted subjects from Wikipedia to increase number of mapping proposals
• Subject based clustering provides
• Extra context, not solely based on textual similarity
• Boost clarity on mapping proposals
• Can increase the number of false positives
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
12/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Update mapping tool- combining different solutions
• Need a normalized total score, so used weighted approach.
• Weights should be configurable• Should depend which metric performs best, e.g. text based matching is better
than context based matching and receives a higher weight…
• But need expert knowledge to make that call
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
13/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Updated mapping tool- Concept relations and flooding (explained)
1. For every proposed mapping, e.g. between two occupations
2. These occupations have relations, like hierarchical or occupation-skill relations
3. Look at ‘nearby mappings’ of concepts connected through these relations
4. If these ‘nearby mappings’ have a high value, ‘flood’ some of this value to the target mapping being considered
5. This is iterative, if the user accepts or rejects a proposal, the certainty of the suggestion changes to 1 (accepted) or 0 (rejected) and the flooding can be redone.
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
14/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Concept relations and flooding❶
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
15/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Concept relations and flooding❷
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
16/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Concept relations and flooding❸
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
17/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Concept relations and flooding❹
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
18/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Concept relations and flooding❺
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
19/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Updated mapping tool –enhanced clarity on mapping proposals
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
20/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Updated mapping tool –enhanced clarity on mapping proposals
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
21/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Updated mapping tool –enhanced clarity on mapping proposals
Reduce uncertainty for proposed mappings
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
22/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Mapping tool – workflow
2 Step approach: first basic mapping, second refined mapping
+ Faster to functional result
+ Earlier gap analysis
+ Allow perfecting the mapping at a later stage
- Good proposals visited twice
• Experts can manually add/correct mappings
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
23/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Mapping tool - general
• Two step approach towards mapping
• Focus on efficiency in the workflow
• Need for usability improvements
• Features to be added (lessons learned):• Function to add additional mappings
• Filter and search capabilities
• Side-by-side concept browsing
• Ignore stop words/words without meaning
• Configuration of scoring parameters
• Automatically check mappings on logical errors
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
24/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Feedback received from participants on data and mapping tool• On the data : expert involvement facilitates
• Validate the mapping outcome
• Assure the quality of the translations (not an objective of the mapping pilot)
• On the process: no comments
• On the mapping tool: overall positive, however:• Approach by the tool differs from current practice
• Need more functionalities in validation interface
• Persist user status in workflow
• Noise level in mapping proposals is sometimes too high
• ISCO filtering: differences due to differences in how concepts are classified (irrelevant when mapping complete classifications)
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
25/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Agenda
Time Agenda item Time
10:00 Opening of the meeting 12:45 General conclusions mapping classifications
10:20 Status update mapping pilot
13:45 Coffee break
11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching
11:15 Country specific conclusions mapping classifications
14:45 Next steps & planning
12:00 Lunch 15:00 End of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
26/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Country specific conclusionclassifications mapping
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
27/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
NOC specific results
• Detailed analysis per NOC
• Numbers represent what can be done purely using tool
• No manual addition of mappings
• TEG completed mappings
• Noise reduction by tool: 90%
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
28/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Common vocabularyParameter Description
Number NOC & NSC Concepts Total number of OCC and KSC from PES to be mapped
Exact Matches Transitive, symmetric, interchangeable
Close Matches Similar, sometimes interchangeable
Broad Match Hierarchical relation
Narrow Match Hierarchical relation
No Match from ESCO (54 – 353) Absolute number of percentage of concepts without match
No Match from NOC Absolute number of percentage of concepts without match
Mandays @ PES # days to complete mapping HOSP
Mandays @ TEG # days to complete mapping HOSP
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
29/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
30/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Mapping results – NL NOC
• Received feedback on the use of ISCO, different views on ISCO tagging resulted in need of manual adding of matches outside of HOSP
• Note this is due to the restriction of the mapping pilot to the HOSP sector, the effect of this restriction on a full mapping should be of lesser impact.
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
31/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
• More NOC concepts, finer granularity
• 50% of ESCO occupations has exact match
• many of NOC occupations are not matched:• Different approach towards cross-sectoral
occupations
• Different interpretation on the scope of the sector (94 are not ESCO HOSP)
• ISCO filtering
• 50% of NOC skills are not matched
• Occupations are broader while skills are narrower
ESCO: 54 OCC & 353 KSCOCC KSC
Number of NOC and NSC Concepts
181 605
Exact Matches 28 94
Close Matches 19 114
Broad Match (ESCO>NOC) 37 41
Narrow Match (ESCO<NOC) 16 285
No Match from ESCO 7 123
No Match from NOC 113/19 362
Mandays @ PES
Mandays @ TEG 0.5 1
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
32/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
33/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Mapping results – FR NOC
Granularity mismatch Granularity mismatch
“fiche métier” (Occupation group) “appélations” (occupations)
Many terms Few terms
Significant words decreased in value Mapping base too small
a balance of both?
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
34/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
• Granularity difference with ESCO• No NPTs at level of appélations• ISCO filtering (364 out of 534
unmatched NOC appélations are not ESCO HOSP)
• 33% additional mapping because subject based clustering (7k)
ESCO: 54 OCC & 353 KSCFROCC
FRKSC
Number of NOC and NSC Concepts
652 374
Exact Matches 2 8
Close Matches 134 64
Broad Match 64 95
Narrow Match 56 507
No Match from ESCO 14 115
No Match from NOC 534/170 187
Mandays @ PES
Mandays @ TEG 1 2
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
35/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
36/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
• Notes:• Missing relation between skills and
occupations
• Missing non-preferred terms
• Most ESCO occupation mapped, many with exact match
• 70% of NOC skills are mapped, but many ESCO skills are not
ESCO: 54 OCC & 353 KSC OCC KSC
Number of NOC and NSC Concepts
115 108
Exact Matches 34 9
Close Matches 40 72
Broad Match (ESCO>NOC) 37 111
Narrow Match (ESCO<NOC) 13 44
No Match from ESCO 4 227
No Match from NOC 69 39
Mandays @ PES
Mandays @ TEG 0.5 1
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
37/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
38/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
• High number of unmatched concepts• ISCO filtering
• Solr rules to be improved
• Mappings done by PES, more restrictive
• Low compatibility of skill descriptions
CZ OCC CZ KSC
No. Nat.Class. Concepts
45 303
Exact Matches 0 101
Close Matches 17 63
Broad Match 4 93
Narrow Match 12 79
No Match from ESCO (54 OCC– 353 KSC)
31 182
No Match from NOC 22 157
Mandays @ PES
Mandays @ TEG 0 0
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
39/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
40/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Agenda
Time Agenda item Time
10:00 Opening of the meeting 12:45 General conclusions mapping classifications
10:20 Status update mapping pilot
13:45 Coffee break
11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching
11:15 Country specific conclusions mapping classifications
14:45 Next steps & planning
12:00 Lunch 15:00 End of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
41/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
General conclusionsclassification mapping
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
42/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Agenda
Time Agenda item Time
10:00 Opening of the meeting 12:45 General conclusions mapping classifications
10:20 Status update mapping pilot
13:45 Coffee break
11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching
11:15 Country specific conclusions mapping classifications
14:45 Next steps & planning
12:00 Lunch 15:00 End of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
43/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Status update job vacanciesand curriculum vitae matching
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
44/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
• Tagging
• Scenario’s
• Evaluation grid
CV-JV Matching
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
45/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Task Who Status
Selection of CV/JV TenForce Done
Translation of CV/JV TenForce Done
Annotation of CV/JV PES In progress
Evaluation grid TenForce To do
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
46/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
47/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Matching scenario’s
• + additional trials to investigate effects and parameters
1 Input Actor Output MATCH Input Actor Output
CV language A PES A CV tagged PES A JV language B PES B JV tagged PES B
CV tagged PES A MAP CV tagged ESCO
CV tagged ESCO MAP CV tagged PES B
2 Input Actor Output MATCH Input Actor Output
CV language A PES A CV tagged PES A JV language B PES B JV tagged PES B
CV tagged PES A MAP CV tagged ESCO JV tagged PES B MAP JV tagged ESCO
3 Input Actor Output MATCH Input Actor Output
CV language A TF CV tagged ESCO JV language B TF JV tagged ESCO
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
48/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Agenda
Time Agenda item Time
10:00 Opening of the meeting 12:45 General conclusions mapping classifications
10:20 Status update mapping pilot
13:45 Coffee break
11:00 Coffee break 14:00 Status update job vacancy, curriculum vitae matching
11:15 Country specific conclusions mapping classifications
14:45 Next steps & planning
12:00 Lunch 15:00 End of the meeting
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
49/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
Next steps and planning
31/07/2015
ESCO - Eurpean Skills, Competences and Occupations
50/
ESCO - European Skills, Competences, Qualifications and Occupations November 3, 2015
•Homework: processing the CVs and JVs • Before Fri 13 November 2015
•Workshop Tuesday 15 December 2015• Mapping• Matching results : grid• Actions• Conclusions
Next steps and planning