A web-based Census of services:

an ISTAT evolutionary study

Annalisa Cesaro, Bernardo Palazzi,

Manlio Paterniti, Paola Ranaldi

5/03/2013

1. The Italian experience in numbers

2. The Italian experience: the web architecture

3. The census web technologies

4. The Census Management System (CMS): a metadata-driven system

5. CMS: a functional overview

6. CMS: building blocks for a cost effective improvement

7. What is SOA?

8. SOA as a Web Service Architecture

9. SOA as a Web Service Architecture in Census LAN

10. SOA criticalities

11. SOA-based web solution ISTAT exploration: technical feasibility

12. SOA-based web solution ISTAT exploration: service overhead

13. SOA-based web solution ISTAT exploration: network latency

14. Conclusions

CENSUS TODAY

SOA PROPOSAL

ISTAT STUDY

2010-2012 online Census Surveys:

• 60 thousand forms for 2010 Agricolture Census (3%)

around 7000 census operators

• 8,5 million forms for 2011 Population and Household Census (one third

directly filled by citizens) (33%)

around 100000 census operators

• 420 thousand forms for 2012 Industry and Services Census (54%)

around 3000 census operators

The Italian experience in numbers

The Italian experience: the web architecture

Census network

internet

internet

Census Operations

Management System (CMS)

Online questionnaire (OQ)

Documentation System (DOC)

Oracle

MySQL

The census web technologies

MySQL

GUI QO

Actions

Services

DAOs

Entities

Not-SOA

GUI CMS

Actions

Services

DAOs

Entities

Web site

Typo3

PHP

Oracle

Process for Postal System on its

own IT

Microdata Processing for

dissemination on its own

IT

Microdata Processing for

first evaluations on its

own IT Replicated

mappings

Replicated

mappings

connection

Replicated

mappings

connection

or BATCH

sharing of

data

Replicated

mappings

connection

Replicated

mappings

connection

or BATCH

sharing of

data

The Census Management System: a metadata-driven

system

• Text are stored as metadata;

• The questionnaire processing rules are stored in metadata;

The questionnaire processing has been modeled as a finite state

machine. A triggering event determines a questionnaire state transition.

The state transition tables are a key element in CMS database.

1 | delivery 31 | arrival

registered

arrival registration

function (function 1)

Function 1 Function 2

State 1 State 31 State 1

State 2 State 2 State 2

Output

State 1 Delivery

… …

State 31 Arrival

registered

The Census Management System: a functional overview

A Census Management system:

• manages census operators: creation of the operators networks (roles

and responsabilities);

• eventually integrates different data sources for minimizing errors;

• trace the questionnaires life cycle, by handling the state flow of each

questionnaire;

• monitoring of the survey process by supervising of all the operations

performed on each questionnaire; SEPARATE TABLES –

UPDATED PERIODICALLY –

HEAVY PROCESS

BATCH PROCESSING FOR

UPLOADING AND

VALIDATING IMPORTED

DATA FROM DIFFERENT

SOURCES – HEAVY

PROCESS

The Census Management System:

building blocks for a cost effective improvement

A WEB BASED CENSUS OF SERVICES WHICH IS

BETTER CONNECTED TO BOTH INSIDE AND OUTSIDE

BETTER HEAVY PROCESSES ON

DEDICATED DB SERVERS FOR

PERFORMANCE

BETTER NONE CODE REPLICATION

FOR MAINTENANCE BETTER EASILY INTEROPERABLE

SYSTEMS

STRONG METADATA - DRIVEN

APPROACH

WELL DEFINED FUNCTIONAL

PROCESSES

What is SOA?

SOA

Front

end Services

Service

Repository Service

Bus

Implementation Interface

Busines

s

Logic Data

GUI

Actions

Services

DAOs

Entities

Not-SOA

Web

Application

…

NO CODE

REPLICATION

INTEROPERABILITY

TECH NEUTRALITY

SOA as a Web Service Architecture

“A Web Service is a software system designed to support interoperable

machine-to-machine intercation over a network. It has an interface

described in a machine processable format (specifically WSDL). Other

systems interact with the Web Service in a manner prescibed by its

description using SOAP messages, typically transported on HTTP with an

XML serialization.” w3c specification

Security Reliability Transaction

messaging

XML

M

E

T

A

D

A

T

A

SOA as a Web Service Architecture in Census LAN

DB

Web Service CLIENT

REQUEST/RESPONCE

SOAP/XML

Container

Web Service SERVER

DATA & METADATA

DATA & METADATA

SOA

Front end –

Web service

CLIENT

Web

Service

SERVER

Service

Repository Service Bus

SOA criticalities Distributed objects have a number of architectural

challenges and others point out:

• Problems introduced by latency and unreliability of

underlying transport;

• The lack of shared memory between the caller and

object;

• The problems introduced by partial failures

scenarios;

• The challenges of cuncurrent access to remote

resources;

• Overhead in terms of serialization/deserialization

and service handle ;

• Great amount of data;

NET LATENCY

EXCEPTION

CLASSIFICATION

PERFORMANCE

TECHNICAL FEASIBILITY

IN CENSUS CONTEXT

Client

program

WS handle

XML SOAP

message

serialize deserialize

XML SOAP

message

network

XML SOAP

message

XML SOAP

message

deserialize serialize

SOA-based web solution ISTAT exploration:

technical feasibility

Oracle

GUI

Actions

SOA

Web

Service

Services

DAOs

Entities

Menu item

Function 1

Function 2

Function 3

Function 4 1:1 RELATIONSHIP WITH

WEB SERVICE OPERATION TECHNICAL FEASIBILITY

IN CENSUS CONTEXT

WAR file

WAR file

SERIALIZABILITY

THROUGH EXTERNAL

JAX-WS BINDING

NO CIRCLE REFERENCES

IN ENTITIES

SOA-based web solution ISTAT exploration:

service overhead Machine A

Front end WAR file

Web Service WAR file

Oracle

Machine B

NO NETWORK LATENCY

TESTS UP TO 10000

RECORD EXCHANGED

SOA-based web solution ISTAT exploration:

network latency Machine A

Front end WAR file

Oracle

Machine C

YES NETWORK

LATENCY

Machine B

Web Service WAR file

TESTS UP TO 10000

RECORD EXCHANGED

Conclusions

- First classification of advantages of a

SOA-based web architecture in survey context,

thanks to the metadata-driven features of actual

web-based systems and to the modular

functional nature of them;

- Piloting applications for:

- technical feasibility testing;

- first classification of SOA

criticalities in census context;

- Performance evaluation for

design/monitoring/routing

purpouses;

GOOD RESULTS

PRODUCES FURTHER

EXPLORATION NEED!

Thank you