© 2010 Microsoft Corporation. All rights reserved.
Quality Assurance: Towards Tools for Characterizing and
Comparing Digital Documents
Natasa Milic-FraylingMicrosoft Research Cambridge UK
www.planets-project.eu
© 2010 Microsoft Corporation. All rights reserved.
What is the problem?
• Digital is a victim of its own success
i.e., the advances in digital technologies that make digital media broadly used and adopted
Document formats, software and hardware are becoming obsolete faster than we can ensure the forward compatibility of the content.
© 2010 Microsoft Corporation. All rights reserved.
What are technical solutions?
• We have two main strategies:
emulation and simulation– Create emulators of hardware
and simulators of software systems to enable old programmes to run and old data to be used.
content migration– Migration to
standards that are likely to be supported in the future.
© 2010 Microsoft Corporation. All rights reserved.
Preservation and
Long-term
Access through
NETworked
Services
• Ensure long-term access to Europe’s cultural and scientific heritage• Improve decision-making about long
term preservation• Ensure long-term access to valued
digital content• Control the costs through automation,
scalable infrastructure• Ensure wide adoption across the user
community• Establish market place for preservation
services and tools
• Build practical solutions• Integrate existing expertise, designs
and tools• Share and build
© 2010 Microsoft Corporation. All rights reserved.
The British Library
National Library, Netherlands
Austrian National Library
State and University Library, Denmark
Royal Library, Denmark
National Archives, UK
Swiss Federal Archives
National Archives, Netherlands
Hatii at University of Glasgow
University of Freiburg
Technical University of Vienna
University at Cologne
Tessella Plc
IBM Netherlands
Microsoft Research, Cambridge
ARC Seibersdorf research
PLANETS Partners
© 2010 Microsoft Corporation. All rights reserved.
Microsoft & PLANETS: Preserving Office Documents• Microsoft Research role within PLANETS:
– Conversion of binary Microsoft Office Documents into Office Open XML File Format (OpenXML)
• We extended the effort to include other formats– More legacy formats, e.g. WordPerfect– Other open standards, e.g. Open Document Format.
Binary MS Office OpenXML
WordPerfect ODF
Binary MS Office OpenXML
DOS Word UOF
© 2010 Microsoft Corporation. All rights reserved.
Document Conversion Tools – Our Approach
• Three-step approach, resulting in a modular and extendible infrastructure– Identify existing conversion tools and libraries– Wrap these tools and libraries into re-usable components– Integrate these components into PLANETS and other
systems.
• If possible, do not use the office applications (e.g., Microsoft Office or OpenOffice.org)– They are designed as interactive applications– Message boxes might pop up (“Do you want …”)– Unclear license question when running on a server.
© 2010 Microsoft Corporation. All rights reserved.
Reusable Components
Transformer Box (Wrapper)
“Binary OpenXML”TB
Interface
Watch Folder Tool
Web Service
ToooXML (GUI)
© 2010 Microsoft Corporation. All rights reserved.
Extendible Architecture
Transformer Box (Wrapper)
“ODF OpenXML”
Transformer Box (Wrapper)
“WP OpenXML”
Transformer Box (Wrapper)
“Binary OpenXML”TB
Interface
Watch Folder Tool
Web Service
ToooXML (GUI)
© 2010 Microsoft Corporation. All rights reserved.
More Technical Details (1)
• Currently two types of wrappers for– Command-line tools (stand-alone executables)• OpenXML/ODF Translator (OpenXML ODF)• OpenXML Document Viewer (OpenXML HTML)
– Microsoft conversion libraries (CNV libraries)• WordPerfect RTF• RTF OpenXML• …
• We allow wrappers to be chained– WordPerfect RTF OpenXML ODF.
© 2010 Microsoft Corporation. All rights reserved.
Microsoft Word
More Technical Details (2)• Microsoft conversion libraries (CNV libraries)– Originally designed to import/export “foreign”
document formats into/from Microsoft Word– Based on the Microsoft Conversion API• Foreign2RTF• RTF2Foreign
– Transformer Box CNV Wrapper follows this API.
Transformer Box CNV Wrapper
CNV LibraryRTF2Foreign Foreign2RTF
© 2010 Microsoft Corporation. All rights reserved.
Supported Formats Source formats
WordPerfect 5 WordPerfect 6 DOS Word Word 2, 6, 95 Word 97-2003 RTF ODF OpenXML
Target formats OpenXML ODF UOF HTML XCDL (format defined in
PLANETS/PC)
© 2010 Microsoft Corporation. All rights reserved.
SIMILARITY ASSESSMENTunderstanding the quality criteria
© 2010 Microsoft Corporation. All rights reserved.
How do we explore and compare digital artefacts
• Perceptive aspects of the digital object– In the past printed version of the document and
screen display
• Interactive aspects of the digital objects– Dynamic content includes both individual artefacts
and the `stream characteristics‘.
• Non-perceptive aspects of the digital objects– Document object model, cashed data, action
generated metadata, hidden formulas, etc.
© 2010 Microsoft Corporation. All rights reserved.
EXAMPLE: Perceptive features for Word Documents
• Two objects in different formats are mapped onto the normalize form– E.g., a WP file converted into .docx. For both we
create an XPS representation of the document
• Feature extraction and comparison– For each feature develop a `digital object probe‘ that
extract the feaeture and measure a property of the feature
– E.g., pass XPS through OCR package and extract various layout features.
© 2010 Microsoft Corporation. All rights reserved.
What is ahead of us?• Research
– What is the relationship between the human criteria and automated measurements? What usage scenarios do we aim for?
• Technology– What ‘instruments’ do we need to extract and measure properties
of the digital content? – How do we automate the process of inspection and quality
assurance?• Legal
– How do we run legacy software as services? We need updated licensing agreements.
– How to provide services that combine open source and non-open source software?
© 2010 Microsoft Corporation. All rights reserved.
THANK YOUContact: Natasa Milic-FraylingMicrosoft Research [email protected]