WDL Technical Architecture Working Group (TAWG)
June 2010
Achievements and Recommendations
Co-chaired by Noha Adly, Bibliotheca Alexandrina Babak Hamidzadeh, Library of Congress
TAWG• TAWG is responsible for establishing policies
and guidelines needed to govern the technical aspects of the WDL.
• TAWG will advise the WDL Project Manager and the future Executive Committee on matters relating to – Architecture– Workflow– Standards– Site functionality– Technology transfer
Meetings- April 2007
8 Members• UNESCO• National Library of China• National Library of Russia• Mexico• Bibliotheca Alexandrina• Library of Congress• National Library of the Netherlands• Germany
Meetings- April 2007• Defining the objectives and the target audience of WDL• Discuss the merits of a centralized vs a portal model • Discuss workflow models• Define the main aspects of WDL website (Connectivity,
access points, content, search, etc.)• Types of content and ingestion models• Presentation and usage of the website• Intellectual added value• Access (multilingual search and display)• Context (narrative and interpretive content)• Participation (inclusion of social networking)
Members in December 200918 Members:• Bibliotheca Alexandrina• Brown University Library• King Abdullah University of Science and Technology• Central Library, Qatar Foundation• Italian Ministry for Cultural Heritage and Activities• Library of Congress • National Diet Library• National Library of Bulgaria• National Library of China• National Library of Finland• National Library of France• National Library of Israel• National Library of the Netherlands• National Library of Sweden• UNESCO• University of Pretoria• Yale University• National Library of Serbia
The WDL Architecture Overview
The WDL Content Pipeline
The WDL Catalog Application
The WDL Web Application
Akamai NetStorageAkamai NetStorage
Akamai Global Edge NetworkAkamai Global Edge Network
http://www.wdl.org
LC ServerLC Server
Solaris 10Solaris 10
MySQLMySQL JavaJava PythonPython File SysFile Sys
LuceneLucene DJangoDJango
nginxnginxSolrSolr ApacheApache
memcachedmemcached
SquidSquid
Uncached Searches
All Item ContentStatically Rendered HTML
Recommendations - Technical• Ensure the WDL website has the following features
– Reliability– Scalability, especially with the increase amount of content– Has multiple access points– Powerful searching tools
• Multilingual• Metadata and content
– Deliver services to users with low bandwidth• Provide full-text search on metadata and contents• Enabling bulk transfer of metadata• Automation of fetching objects and data• Bandwidth and the transfer of large files• Improve documentation on digitization
Recommendations - Technical• Content transfer, metadata compliance, translation– Document current practice and desired improvements– Review existing tools and specifications.– Adopt tools and workflow that are scalable
• Improving the cataloguing tools to allow partners editing data
• Develop tools for quality control to ensure consistent use of terminology
• Discussions with Wikipedia for linking individual WDL documents with relevant Wikipedia pages
Recommendations - Content
• Current methods and tools used for transferring content and standardizing metadata needs to be more scalable
• Consistent use of terminology for translation: Cooperation with the Translation Working Group.
• Content selection guidance: Cooperation with the WDL Content Selection Working Group
• Improve documentation on metadata standards
• Asking for richer metadata to enrich functionality in the future
Recommendations- Policy
• Developing requirements and policies for data distribution
• Collaboration between WDL and Europeana
• Extending the languages of the WDL beyond the existing seven
Future Considerations• Preservation– Role of WDL wrt providing preservation services– WDL could undertake a survey of existing preservation
practices among partners.– WDL partners with preservation experience train other
partners to build capacity for preservation and digital conversion.
• Sharing the WDL software through licensing, open source, APIs
• Website features– Mobile devices– Building a user interactive community– Allowing users to use and annotate the content– Display options for comparing items
Future Considerations
• TAWG’s role towards new partners and partners with less experience:– Technology transfer– Training– Adopting a country/partner
• Establishing sub-groups to deal with particular issues
• Sharing relevant experiences – Social networking– Preservation – Metadata– Access
Thank You