CES
IQML overviewand
Metadata in Codacmos
Defeng Ma ([email protected])
Centre for Educational Sociology
The University of Edinburgh, UK
2CES15 October 2003, EuroStat, Luxembourg
Outline
• IQML overview
• IQML QDT (Questionnaire Design Tool)
• IQML SAT (Survey Administration Tool)
• IQML Registry/Repository
• Codacmos project
• Metadata for data collection (WP6 in Codacmos)
4CES15 October 2003, EuroStat, Luxembourg
IQML Partners
• University of Edinburgh, UK – coordinator, QDT
• Comfact, Sweden – QPT, DIT, XML specification
• Desan, Netherlands – SAT
• Dimension EDI, UK – standards, repository
• CSO, Ireland – user
• Statistics Norway – user
• National Technical University of Athens, Greece – user
Funding under EuroStat EPROS as part of the EU Funding under EuroStat EPROS as part of the EU 55thth Framework R&D Framework R&D
5CES15 October 2003, EuroStat, Luxembourg
IQML modules
• QDT(Questionnaire Design Tool): design the questionnaire
• SAT (Survey Administration Tool): manage the survey
• Metadata Repository: storage and retrieval of metadata
• QPT (Questionnaire Presentation Tool): present questionnaire to the user and collection the data
• DIT (Database Interrogation Tool): extraction of data from database to the questionnaire
6CES15 October 2003, EuroStat, Luxembourg
The IQML Processing/Data Flow
Surveyinfo
Surveyinfo
RespondentRespondent
DatasetDataset
qnaireqnaire
IQML QDT IQML SAT
SAT IQML QPT/DITQPT/DIT
IQML Metadata Repository
MetadataRepository
7CES15 October 2003, EuroStat, Luxembourg
System Structure
APIAPI Metadata
Repository
Questionnaire Designer
API API
DatabaseIntegration Tool
SurveyAdministration
Questionnaire Presentation
QuestionnaireXML
Common Model
•Questionnaire presentation (QPT)
•Database interrogation (DIT)
•Shareable metadata
•Published questionnaire
Interoperability, Technical Infrastructure and Standards
MetadataRepository
Web Application
•Questionnaire designer (QDT)
•Survey administration (SAT)
•Externalisation (e.g.IQML XML form)
ApplicationLocal Storage
EEG6/WG4 adopted (as basis for XML4DR)
Syntax is XML
Storage
API
Repositpry
API
Web Servlet
C++, Java Script, Active X
Java
Common modelQuestionnaire
id : String
Set, Tuple, Data, Format
order of Data must be same as order or Domain
Code
codeValue : String
Questionnaire
id : String
Domain
domainId : String
Set
setId : StringsetType : Stringstate : String
Rendering
renderingType
CodeList
codeListId : String
Data
dataId : String
Tuple
tupleId
DataType
type : String
Decimals
decimals : String
Format
formatId : String1..n
1
1
0..n0..n
<<Ordered>>
0..1
1
1..n
1
0..n0..n
1
0..1
0..n1 0..n
0..n
0..n0..n
<<Ordered>>
0..n0..n
0..10..1
1 0..11
0..n
0..1
0..n
0..10..1
0..n
0..10..1
0..10..1
IQML Common Model used in MR
11CES15 October 2003, EuroStat, Luxembourg
Models mapping forInteroperability
• QDT model -> IQML common model (High level only)» Questionnaire, questionnaire body -> Questionnaire
» Section, Question -> Set
» Response Group -> Domain
» Variable Group (Sub question) -> Tupel
» Variable -> Data
» Member of Response Group -> Code List, Code
• This mapping is used to publish questionnaire from QDT to Metadata repository
12CES15 October 2003, EuroStat, Luxembourg
Mapping to external standards: DDI (in Cosmos project)
Surveyinfo
Respondent
Dataset
qnaire
IQML QDT
IQML SAT
IQML QPT/DIT
IQML Metadata Repository
FasterNesstar publisher
CosmosRegistry
Catalogue.xml
Catalogue.xml Searching
URL of DDI object
FasterSurvey
FasterNesstarServer
(UKDA)
Publishing
QnaireDDI
UEDINWeb
Server
SurveyDDI
DESANWeb
Server
14CES15 October 2003, EuroStat, Luxembourg
Philosophy of QDT
question sectionquestionquestionquestionquestion
sectionsectionsectionsection
NotesGroup
elementsNotesNotesNotesNotes
Group elements
Group elements
Group elements
Group elements
DefineQuestion type
Create basic components
Question bank
Assemble questionnaire
Publish questionnaire
15CES15 October 2003, EuroStat, Luxembourg
Defining Question Type
Question type defines • data type (text / integer / real / boolean)• response type (simple / multi / choice)• sub-questions (yes / no)
16CES15 October 2003, EuroStat, Luxembourg
Main QDT components
• Question type
• Question bank
» Element groups
Response Group
Missing value group
Sub-Question group
» Note
» Question
» Section
• Control Elements
» Calculation
» Validation
» Navigation Rule (‘Dynamic Guidance’)
• Questionnaire
17CES15 October 2003, EuroStat, Luxembourg
QDT – complex question
Sub-question group
Question text Response group
variablevariable group
19CES15 October 2003, EuroStat, Luxembourg
• Sampling import population database
select sample(s)
• Survey definition questionnaire (QDT -> Reg/Repository -> SAT)
sample(s) (SAT -> Reg/Repository -> SAT)
• Distribute survey• Monitor response• Send reminders
Functions
20CES15 October 2003, EuroStat, Luxembourg
Sample & Survey Process
Questionnaire(QDT)
txt xsl
Pop
sample Sample Survey
(RR)
21CES15 October 2003, EuroStat, Luxembourg
Distribution & Monitor Process
Survey
Message
•Html generation•Prefill•Distribution
Response(QPT)
Monitoring
Reminder
23CES15 October 2003, EuroStat, Luxembourg
IQML Reg/Rep
• ebXML standard
• Web servlet Interface with Tomcat web server » Remote accessible
• Administration GUI
• Not directly visible for the user, the QDT and SAT can communicate through the Reg/Rep
• IQML common model for storing objects in the repository
24CES15 October 2003, EuroStat, Luxembourg
Contact Details for IQML
Defeng Ma ([email protected])
http://www.epros.ed.ac.uk/iqml
25CES15 October 2003, EuroStat, Luxembourg
IQML -> Codacmos
• IQML is mainly focus on the primary data collection based on the web/email survey (Trails in SSB, CSO and NSSG)
• Codacmos (Cluster of Data Collection Integration & Metadata Systems for Official Statistics ) is a cluster of relevant projects» TELER,
» DATAMED,
» IQML and
» IPIS
27CES15 October 2003, EuroStat, Luxembourg
Codacmos: Goal
• Main Goal is to identify ways » improve the cost-effectiveness of the data collection
process and the quality of official statistics, and» lowering the statistical burden of the respondents
28CES15 October 2003, EuroStat, Luxembourg
Codacmos: Workpackages
• WP1: Management and Coordination
• WP2: Integration of available results
• WP3: Primary data collection
• WP4: Secondary data collection
• WP5: Integration of primary and secondary data collection
• WP6: Metadata system for data collection
• WP7: Relevant issues for FP6
• WP8: Assessment, Evaluation and Peer review
• WP9: Dissemination and Publication
30CES15 October 2003, EuroStat, Luxembourg
Data collection
NSI
Market ResearchInstitute
TaxOffice
Respondent
Respondent
Respondent
Respondent
31CES15 October 2003, EuroStat, Luxembourg
Data sharing
Data Provider
Data Provider
Data Provider
NSI
EuroStat
World Bank
32CES15 October 2003, EuroStat, Luxembourg
Optimum solution?
Virtual Metadata
Repository
Input OutputData Provider
Data Provider
Data Provider
NSI
EuroStat
World Bank
33CES15 October 2003, EuroStat, Luxembourg
Scope of the WP6 model
DisseminationProductioncollection
DataDefinition of statistics
Statistical production process*
Focus on Data collection, Thinking of the whole processBut: where to stop?
* Picture from H. Rouhurirta: “On the structing of statistical Information”, First Metanet conference, 2001
34CES15 October 2003, EuroStat, Luxembourg
Metadata for data collection
Primary Data Collection
Secondary Data Collection
Integration of primary and secondary
Processing metadata
StatisticalData
Content Metadata
• Metadata for definition statistics (survey, questionnaire, statistical unit, etc.)
• Metadata for production (disclosure, quality control, constrains, etc.)
Additional:
35CES15 October 2003, EuroStat, Luxembourg
Looking back …
• State-of-art review
• Existing systems/models review and analyse
• Study» Similarity/Common elements
» Difference
» Pro/Con
• Archive the common understanding of the requirements/user needs
36CES15 October 2003, EuroStat, Luxembourg
Top-down approach for Common metadata model
• Develop the Abstract model at top level first to cover the common aspect of the metadata for data collection process (cover the large scope)
• Then go down in the low levels with more details of the metadata models (few examples) as far as we can:» Statistical data (content)
» Processing
» Constraints
» Survey/questionnaire
» …
37CES15 October 2003, EuroStat, Luxembourg
Working with the top down
• Generalise the data collection process
• Abstract metadata model for data collection
• Define the level of the metadata
• Define the category of the metadata
• Define the exchange model of the metadata
• Define common terminology
38CES15 October 2003, EuroStat, Luxembourg
The abstract model
• Keep the model» Generic
» Simple
» Clear
39CES15 October 2003, EuroStat, Luxembourg
Bottom-up approach: Start with what we have…
• Metadata in the view of different aspect» Primary/Secondary data collection
» Organisation/International
» Enterprise/individual
» Economical/Social
» Administrative/statistical data
» Centralised/distributed
» Register/Survey
» Questionnaire/processing/transformation, etc.
40CES15 October 2003, EuroStat, Luxembourg
Integration/Combination of top down and bottom up
• Mapping existing models to common model
• Identify the tasks for integration
• Outline the working direction/guideline for improve the existing systems
• Refine the common model
• Specify/define the demo experiment field and strategy of the demo
41CES15 October 2003, EuroStat, Luxembourg
WP6 work plan
• Before 1st TM: » preparation, collection, review of state-of-art
• 1st TM (Sep. 2003): » Get common understanding on the target and tasks
» Detailed working plan
» Draft structure of the common model
» Initial discussion of the Integration analyse
• After 1st TM: » common core model
» integration analyse of the existing system
» Demo definition and specification
42CES15 October 2003, EuroStat, Luxembourg
Work plan (cont.)
• 2nd TM (Jan. 2004)» Finalize common model and integration of existing system
» Demo preparation
» Workshop preparation
» Deliverables preparation
• After 2nd TM» Workshop
» Demo
» Deliverables
43CES15 October 2003, EuroStat, Luxembourg
Codacmos website
http://www.codacmos.eu.org
Project coordinator: Alberto [email protected]
Contact Author: [email protected]