University of the Aegean – Department of Information and Communication Systems Engineering

A methodology for Evaluating PSI e-Infrastructures based on Multiple Value

Models

Charalampos Alexopoulos, cPhDEuripides Loukis, Associate Professor

INTRODUCTIONe-Science: cross border research collaboration and

use of huge high-capacity computing resources – tools for collection, storage, analysis and modeling of data

large amount of data is very useful for conducting scientific research in many areas

socio-economic benefitsunderstanding of social and economic problems, and

also of the effectiveness of various policies government agencies implement for addressing them

opening up this resource could amount to about € 40 billion a year in the EU – a new e-market governments start to invest on

5/10/20122 PCI2012 - University of the Aegean

PURPOSEa systematic evaluation of these PSI e-

Infrastructures, aiming at a better understanding and assessment of value they generate

a structured and comprehensive evaluation methodology is missing

“ The proposed methodology includes initially the definition of one value model for each stakeholder group, which consists of the main dimensions of value the PSI e-infrastructure generates for it, and also the connections among them”

5/10/20123 PCI2012 - University of the Aegean

BACKGROUND

Scoping eInfrastructuresStakehol

ders Data

Acquisition Data

ProvisionCommunic

ation

Literature ReviewIS

Evaluation TAM IS Success Models E-Services

5/10/20124 PCI2012 - University of the Aegean

Research Streams InsightsIS EvaluationIS’s offer various types of benefits, both financial

and non-financial, and also tangible and intangible ones, which differ among the different types of IS

it is not possible to formulate one generic IS evaluation method, which is applicable to all IS

a comprehensive methodology for evaluating a particular type of IS should include evaluation of both its efficiency and its effectiveness, taking into account its particular characteristics, capabilities and objectives

5/10/20125 PCI2012 - University of the Aegean

Research Streams InsightsTAM

identify the characteristics and factors affecting the attitude towards using an IS, the intention to use it and finally the extent of its actual usage

perceived usefulness and perceived ease of use determine an individual's intention to use a system with intention to use serving as a mediator of actual system use

IS Success ModelsIS evaluation should adopt a layered approach based

on the above interrelated IS success measures (information quality, system quality, service quality, user satisfaction, actual use, perceived usefulness, individual impact and organizational impact) and on the relations among them

5/10/20126 PCI2012 - University of the Aegean

Research Streams Insightse-Services Evaluationframeworks that assess the quality of the

capabilities that the e-service provides to its users

frameworks that assess the support it provides to users for performing various tasks and achieving various objectives, or users’ overall satisfaction

the above frameworks do not include advanced ways of processing the evaluation data collected from the users, in order to maximize the extraction of value-related knowledge from them

5/10/20127 PCI2012 - University of the Aegean

Our Approach based on Value Models

Ease of Use Experience

Performance

Data Search & Download Capabilities

Data Provision Capabilities

Accompl. UsersObjectives

Use

FutureBehaviour

Users’ Data Analysis Capabilities

Efficiency Level Effectiveness Level

Fut. Behavior Level

Data Users

5/10/20128 PCI2012 - University of the Aegean

Our Approach based on Value Models

Efficiency Level Effectiveness Level

Fut. Behavior Level

5/10/20129 PCI2012 - University of the Aegean

Data ProvidersEase of Use Experience

Performance

Providers’ Data Analysis Capabilities

Data Upload Capabilities

Accompl. ProvidersObjectives

Use

FutureBehaviour

Value MeasuresThe total of 51 value measures were

defined15 common value measures22 value measures for users14 value measures for providers

These value measures was then converted to a question to be included in questionnaires to be distributed to stakeholders

A five point Likert scale is used to measure agreement or disagreement

2 Questionnaires have been formulated5/10/201210 PCI2012 - University of the Aegean

Value Model Estimation Algorithm1. For each value dimension a composite variable is calculated

as the average of its individual measure variables.2. Average ratings are calculated for all value dimensions

(using the composite variables calculated in step 13. For each value dimension of the first level we calculate its

correlations with all value dimensions of the second and the third levels (using again the composite variables calculated in step 1).

4. Combination of 2 classes of analytics calculated in steps 2 and 3 for the construction of a high-level value model of the PSI e-Infrastructure

5. First Layer Value Dimensions Classification into four groups: low rating – high impact low rating – low impact high rating – high impact high rating – low impact

6. Finally we repeat stages 2, 3, 4 and 5, but this time for the individual value measures/variables instead of the value dimensions’ composite variables.

5/10/201211 PCI2012 - University of the Aegean

Conclusions

5/10/2012PCI2012 - University of the Aegean12

This paper has presented a methodology for evaluating an emerging class of IS: the PSI e-Infrastructures.

These IS aim to support the evaluation ofgovernment agencies for opening their data, in order to be

used for scientific, commercial and political purposesvarious groups of users interested in them (e.g. scientists for

conducting research, active citizens and journalists for drawing conclusions on previous government activity)

The proposed methodology assesses a wide range of types of value generated by PSI e-Infrastructures for these two stakeholders’ groups

An algorithm for advanced processing of stakeholders’ evaluation data, which results in the estimation of value models for these two groups and the identification of improvement priorities

References

5/10/2012PCI2012 - University of the Aegean13

Commission of the European Communities (2009), “Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions – ICT Infrastructures for e-Science”, COM (2009) 108 Final, Brussels.

Commission of the European Communities (2011), “Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions – Open data: An engine for innovation, growth and transparent governance”, COM (2011) 882 Final, Brussels.

Loukis, E. Pazalos, K. Salagara, A. (2012), “Transforming e-services evaluation data into business analytics using value models”, Elsevier, Electronic Commerce Research and Applications 11 (2012), 129–141.

Makx Dekkers, Femke Polman, Robbin te Velde and Mark de Vries, MEPSIR – Measuring European Public Sector Information Resources: Final report of study on exploitation of public sector information – benchmarking of EU framework Conditions, report for the European Commission, June 2006

Kim, Y. and Crowston, K. (2011), Technology adoption and use theory review for studying scientists' continued use of cyber-infrastructure. Proc. Am. Soc. Info. Sci. Tech., 48: 1–10. doi: 10.1002/meet.2011.14504801197

Stockdale, R., and Standing, C. (2006), “An interpretive approach for interpreting information systems: a content, context, process framework”. European Journal of Operational Research, 173, , 1090–1102.

Irani, Z., and Love, P., (2008), “Information systems evaluation – a crisis of understanding” In Z. Irani and P. Love (eds.), “Evaluating Information Systems – Public and Private Sector”, Butterworth-Heinemann, UK.

Schepers, J. and Wetzels, M. (2007). A meta-analysis of the technology acceptance model: Investigating subjective norm and moderation effects. Information & Management, 44, pp. 90-103.

Fishbein, M., & Ajzen, I. (1975). Belief, Attitude, Intention, and Behavior. Reading, MA: Addison-Wesley.

Lu, J., and Zhang, G. (2003), “Cost benefit factor analysis in e-services”, International Journal of Industry Service Management, 14, 5, 570–595.

Fassnacht, M., and Koese, I. (2006), “Quality of electronic servic

DeLone, D.H., McLean, E. R. (2003), “The DeLone and McLean model of information systems success: a ten-year update”, Journal of Management Information Systems, 19, 4, , 9–30

Thank you for your attention

5/10/2012PCI2012 - University of the Aegean14