Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLC The Business
Of OrganizedInformation
Metadata Maturity Survey Findings
Taxonomy Community of Practice Call SeriesJuly 15, 2009
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Presenters
• Paul Wlodarczyk– Director, Solutions Consulting, Earley & Associates
• Ron Daniel– Principal, Taxonomy Strategies LLC
2
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Housekeeping
• Calls last 60 minutes
• We will take questions at the end of the call
• Lines will be muted during the call but you can email [email protected] or Skype Rebecca.M.Allen with immediate questions
• Use *6 to mute/un mute your phone
3
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLC The Business
Of OrganizedInformation
Metadata Maturity:Benchmarking Best Practices Survey Results, Analysis, and Conclusions
Paul WlodarczykDirector, Solutions ConsultingEarley & Associates
13 July 2009
Ron DanielPrincipalTaxonomy Strategies LLC
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Introduction
• Presentation is about the results of the 2009 survey of practices in Search, Metadata, Taxonomy, and Information Development.
• Audiences:– Survey participants: See the results of the survey and
how your organization compares to others.– Organization information specialists: See how to
benchmark your organization against others of your size and/or industry. Use that to justify catching up, or taking the next step, with your management.
– Information scientists: See the current state of practice and how it varies from previous surveys, or by size and/or industry of the organization.
5
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Background – Metadata Best Practices Survey• Goals for 2004, 2005 metadata best practices surveys:
– Help develop a Metadata Maturity Model (MMM)– Keep the MMM “tied to reality” through best practices
• MMM was a potential method to:– Identify search and metadata best practices and their
pervasiveness– Predict potential problems in consulting engagements– Help clients identify projects that are within their reach (and that
aren’t)
• In 2009 we updated the survey– Added questions on information development practices (e.g.
content reuse)
• Goals for 2009 survey:– Spot trends in best practices since 2005– Determine which practices impact Findability and Information
Quality– Help validate and refine the MMM
6
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Previous Metadata Maturity Model
7
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
CMMI: The Best-Known Maturity Model
• Developed for Software Engineering, Funded by the DoD.
• ~400 Processes, in 22 Process Areas, Keyed to 5 Maturity Levels.
• Process Areas contain Specific and Generic Practices, organized by Goals and Features, and arranged into Levels
• Process Areas cover a broad range of practices beyond simple software development
• CMMI Axioms:–Individual processes at higher levels are AT RISK from supporting processes at lower levels.–A Maturity Level is not achieved until ALL the Practices in that level are in operation.
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Towards a Metadata Maturity Model
• Caveats:– Maturity is not a goal, it is a characterization of an
organization’s methods for achieving its core goals.– Mature processes impose expenses which must be
justified by consequent cost savings, revenue gains, or service improvements.
• Nevertheless, Maturity Models are useful as collections of best practices and stages in which to try to adopt them.
• We are looking for a model to help clients decide what to do, we are not looking to audit and certify service providers. A simpler model is appropriate.
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Second Metadata Maturity Model (ca. late 2005)
Practice Area Basic Intermediate Advanced Limiting
Search Capabilities • Uniform Search Box• Query Log Examination
• Index Multiple Repos.• Best Bets
• Facet Navigation UI
Metadata and taxonomy standards
• System Metadata Standards• Organization Metadata
Standard
• Multiple Repositories Comply w/ Org. Metadata Std.
• Reuse ERP Taxonomies• Taxonomy Maintenance
Process
• Taxonomy Roadmap• Highly Abstract Subject
Taxos (e.g. “Moods”)• Metadata Maintenance Doc
Tools and tool selection • Requirements, then Tools • Bakeoff Datasets • Budget for Bakeoffs • Tools, then Requirements
Staff training and hiring • Librarian or IA Expertise• Search Analyst Role
• Cross-Functional Taxonomy Creation
• Cross-functional taxonomy maintenance team
• SME Catalogers• Pre-hire Testing
Data creation and QA • CM Introduced • ROT-Elimination• Semi-auto tagging
• Quality Measures
Project management • Project Plan• X-Functional Teams
• Std. Project Methodology• Multi-Year Plan• Communication Plan• SMT Business Manager,
instead of IT Manager
• Early Termination
Executive support and ROI • External Search ROI• SMT in separate silos
• Intranet ROI Model • CEO knows Search ROI • Use it or Lose It Budgets
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Changes to the MMM from ‘04 to ‘05.
• Basic structure simplified from five maturity levels to four.
• Several processes moved up a level of sophistication as evidence showed they were not used at the basic level.
• More practices added, esp. around staff and training.
• The processes and their arrangement were arrived at through a mix of survey results and educated opinion.
• How have things changed in the last four years?
11
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Population statistics
12
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Participants by Organization Size
13
2009 N=2172005 N=60
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Participants by Job Role
14
2009 N=2172005 N=60
52%
39%
23%
39%
115%
100%
44%
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Participants by Industry
15
2009 N=2172005 N=60
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
The State of Best Practices in 2009
16
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Search Practices, 2009 vs. 2005
17
2009 N=1292005 N=54
75%
28%
24%
30%
17%
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Metadata Practices, 2009 vs. 2005
18
2009 N=1272005 N=51
31%
13%
16%
19%
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Taxonomy Practices, 2009 vs. 2005
19
2009 N=1282005 N=52
59%
50%
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Comparing MMM to Reality (1)
• The MMM predicts that practices will be adopted in a certain sequence.– How accurate is that?
• Expect to see percentage not adopting increase as practice is more sophisticated.– See it in search, not in
others.– Overestimated the uptake
of query log examination.
20
Uniform Search Box 14.9
Query Log Examination 35.4
Index Multiple Repositories 28.8
“Best Bets” 39.1
Facet Navigation UI 39.1
System MD Stds. 40.4
Org. MD Std. 39.9
Multiple Repositories Comply 43.8
Reuse ERP Taxonomies NA
Taxo. Maint. Doc. 37.6
Taxonomy Roadmap 36.6
Highly Abstract Vocabularies NA
Metadata Maint. Doc. 73
Search
Standards
Percentage yet to adopt a practice
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Comparing MMM to Reality (2)
• How much growth in maturity is due to 1-person practices vs. group practices?– Solo practices
flagged in yellow.– Most change is due
to group practices.
21
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Information Development Practices, 2009
22
2009 N=1752005 N/A
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Analysis: IM Challenges by Company Size
23
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Key Question: Do mature organizations perform better?• We used the survey responses to develop a
“Metadata Maturity Quotient”– Score on individual survey responses:
• 0 = not practiced• 1 = under development• 2 = practiced in one unit• 3 = practiced in more than one unit
– Separate scores for search, metadata, taxonomy, and information development
– Overall Metadata Maturity Quotient is sum of Search, Metadata, and Taxonomy scores
– Eliminated respondents with large number of “Don’t Know / NA”
24
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Size Guides our Analysis
• Preliminary analysis showed that size matters• Divided respondents into groups by organization
size:– > 10,000 employees– 1,000 – 9,999 employees– 100 – 999 employees– < 100 employees
• Then into quartiles based upon Maturity Quotient:– Best in Class (top quartile)– Majority (middle two quartiles)– Laggards (bottom quartile)
25
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Maturity Quotient Score, by Organization Size
• Maturity increases with larger organizations because of IT investment, need
• Small organizations can get there faster with less complex deployments
• Selection bias – our small companies are more content-intense…
26
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Maturity Quotient Score, by Industry
27
N=128
Knowledge / content-intensive
Product / Technical
Transactional
Operational
Archival
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Respondents by Industry, Org Size
Small firms are more likely to be in content-intensive industries that are more mature with respect to metadata best practices.
28
Ave
rag
e I
nd
ust
ry M
eta
da
ta M
atu
rity
Number of Respondents
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Document Types by Organization Size
Mature firms are more document-intensive
29
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Metadata search, by organization size
Mature firms are more search-intensive
30
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Do mature organizations perform better?
• We assessed responses to “Information Management Challenges” for each group of respondents (by size and maturity level)
• Impact / importance of issues varies dramatically by organization size
• Generally, more mature organizations perform better on issues that matter, across all size groups
31
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
“Significant Impact to the Business”
Larger companies have more intense IM needs
32
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
“Significant Impact to the Business”
<100 Employees
100-999 Employees
1000-9,999 Employees
>10,000 Employees
Mature companies fare better than laggards regardless of organization size
33
0.0%
0.0%0.0%0.0%0.0%0.0%0.0%
0.0%0.0%0.0%0.0%0.0%
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Summary
• Larger companies and those in certain information-intense industries have more pain resulting from managing more unstructured content
• Generally speaking, the bigger the organization, the worse the pain
• Organizations with the most pain have adopted best practices in the areas of information development, search, metadata, and taxonomy
• Our findings show that those who adopt these practices have less pain
34
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Next steps
• Use the findings to refine the Metadata Maturity Model– e.g. Move query log examination out of the basic level, add
spelling correction to the basic level, add more staff practices, etc.
– Is there interest in collaborating with us on the next revision? Contact Rebecca Allen ([email protected]).
• Benchmark your organization– Take the survey and score the practices: 1 for under
development, 2 for practiced in one unit, 3 for practiced in multiple units.
– Sum the scores and compare to others in this presentation.– Is there interest in an online tool for this, or for more in-
depth diagnostic assessments based upon the MMM?
35
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
36
Community of Practice Calls
SharePoint IA Group: http://tech.groups.yahoo.com/group/SharePointIACoP/
Taxonomy Group: http://finance.groups.yahoo.com/group/TaxoCoPSearch Group: http://tech.groups.yahoo.com/group/SearchCoP
Upcoming calls:
• August 5, 2009 – Conducting a Search Audit• September 2, 2009 – DITA• October 7, 2009 – Taxonomy Usability Testing• November 4, 2009 – Developing an Ontology• December 2, 2009 – Applications for Topic Maps• January 6, 2010 – Taxonomy Management
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLCThe BusinessOf Organized
Information
Please fill out the survey that should be in your inbox.
Let us know what topics you are interested in and how we can improve the series.
Seth Earley
www.earley.com
781-820-8080
37
Copyright © 2009 Earley & Associates Inc. And Taxonomy Strategies LLC. All Rights Reserved.
Taxonomy Strategies LLC The Business
Of OrganizedInformation
Questions?