Master Thesis Performance measurement in E …Master Thesis -7- TU/e 3.4 STEP 1: E-PROCUREMENT CHAIN...

Eindhoven University of Technology

MASTER

What you measure is what you geta performance measurement approach for operational control of E-procurement chains

Khadimi, A.L.

Award date:2010

Link to publication

DisclaimerThis document contains a student thesis (bachelor's or master's), as authored by a student at Eindhoven University of Technology. Studenttheses are made available in the TU/e repository upon obtaining the required degree. The grade received is not published on the documentas presented in the repository. The required complexity or quality of research of student theses may vary by program, and the requiredminimum study period may vary in duration.

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

https://research.tue.nl/en/studentthesis/what-you-measure-is-what-you-get(a249d18e-8a26-4938-9aee-eae8c7260bcd).html

What You Measure Is What You Get

A Performance Measurement Approach for

Operational Control of E-procurement Chains

Master Thesis

Business Information Systems

September-2010

Student: A.L (Latif) Khadimi

First supervisor: Dr. S.A. (Samuil) Angelov

Second supervisor: Dr. B.F. (Boudewijn) van Dongen

Third supervisor: Prof. Dr. R.J. (Rob) Kusters

External supervisor: J.M. (Job) van der Sijs MBA

Performance Measurement for E-procurement Chain Management

Master Thesis -2- TU/e

Executive summary

Research question and approach

This study explores operational control of electronic procurement chains through

performance measurement. According to Harink (2003) electronic procurement is the use

of internet technology in procurement process. Van Weele (2003) defines procurement as

‚Obtaining from external sources all goods and services which are necessary for running,

maintaining and managing the company’s primary and support activities at the most

favorable conditions‛.

The application of e-procurement systems involves large underlying information systems

where multiple parties either support these systems or interact with them.

In order to maintain success of e-procurement business, the underlying chain of systems

and parties needs to be managed and controlled. The operational control of the chain is the

primary management level that ensures the business operability by monitoring the

operational activities of the e-procurement system and its belonging chain.

Measuring the performance of the chain provides substantial information of the level of the

operability and quality of the e-procurement chain.

Existing studies measure performance of IT-systems or individual organizational units (i.e.

chain participants) and do not address performance of e-procurement chain as a whole.

Moreover, to the best knowledge of thesis’s author, there is no unique performance

measurement methodology in which the performance of e-procurement system and the

corresponding chain is measured exclusively.

Therefore this study delves into performance measurement in e-procurement chains. The

main question of the thesis is:

“How can the performance of an e-procurement chain be measured in order to provide the chain

management with supportive information about its level of operability?”

The objective of the study is to develop an approach for the performance measurement of e-

procurement chains that contains structured and efficient steps. The study identifies the

necessary constructs of the approach and explains how the approach is composed and

should be applied.

The research is divided into three parts:

A literature review of e-procurement concepts, performance measurement concepts,

and performance measurement approaches are carried out. Obtained knowledge is



further analyzed and discussed for performance measurement of e-procurement

chains.

In the second part of the research, a performance measurement approach that targets

the operational control of e-procurement systems and their chains is designed. The

studied performance measurement methods and techniques are used for the

development of each step of the new approach.

The approach is then evaluated by applying it to an industry case, in the third part of

research.

Results

A process-oriented e-procurement performance measurement (PEPM) approach for

operational control of e-procurement chains is developed. Figure 1 illustrates an overview of

steps in PEPM approach.

Step 1: Analyze E-procurement system and

its chain

Step 2: Determine and Measure E-services

Step 3: Select supporting processes

Step 4: Set process

performance objectives

Step 5: Model process

Step 6: Derive process

performance metrics

Step 4: Verify and validate performance

metrics

Process performance measurement

Figure 1: PEPM approach

The process-oriented aspect results in looking at the e-procurement system and its chain as

set of processes. The developed approach is composed of three phases. The first phase is to

control and measure offered electronic services (e-services) of e-procurement system. The

measurement of the e-services is done through applying the quality of service performance

aspects. The ITIL relevant set of processes and performance metrics that address the quality



of service measurement are identified and used for the application of this phase of PEPM

approach (see figure 2).

Controlling E-procurement

services

Controlling accessibility of

services

Controlling timeliness of

services

Controlling conformance

of services

Controlling reliability of

services

Controlling quality of services

Controlling cost of services

Quality of service processes

Figure 2: Controlling e-services

Obtaining critical set of supporting processes in e-procurement situation is the second phase

of PEPM approach. The set of supporting processes were facilitated by introducing a couple

of selection criteria (e.g. outsourcing). The set of ITIL supporting processes are adapted and

used to apply this phase of framework (figure 3).

Controlling supporting operations

Controlling service design

Controlling service transition

Controlling service operation

Controlling service strategy

Supporting processes

Figure 3: Controlling supporting services

The third phase of PEPM approach introduces a specific process performance measurement

methodic. It involves process performance objectives discovery, process modeling, deriving

performance metrics, and evaluation of performance metrics.



The developed PEPM approach is then evaluated by applying it to an industry case wherein

this thesis was carried out. Based on the approach composition and findings on its

application on an industry case, the approach is then analyzed.

The PEPM approach provides a systematic methodology for obtaining and identifying the

relevant to be monitored processes for the operational control of e-procurement chain. The

approach underlines what areas of chain are critical for the chain operability and is

considered as a practical and efficient tool by providing structural methods for application of

each of its steps.

Direct implication after application of PEPM approach is to identify how performance

metrics data can be collected and to assign target values for identified performance metrics.



Table of Contents

EXECUTIVE SUMMARY 2

1. INTRODUCTION 10

1.1 COMPANY PROFILE 10

1.2 PROBLEM STATEMENT 11

1.3.1 MAIN RESEARCH QUESTION 13 1.3.2 SUB RESEARCH QUESTIONS 13 1.3.3 RESEARCH OBJECTIVE 14 1.3.4 RESEARCH APPROACH 14 1.3.5 DATA COLLECTION METHODS 15

2. LITERATURE REVIEW 17

2.1 ELECTRONIC PROCUREMENT (E-PROCUREMENT) CONCEPTS 17

2.1.1 PROCUREMENT 17 2.1.2 E-PROCUREMENT 18 2.1.3 E-PROCUREMENT FORMS 19 2.1.4 E-MARKETPLACES 20 2.1.5 E-PROCUREMENT CHAIN AND MANAGEMENT 22

2.2 PERFORMANCE MEASUREMENT CONCEPT 23

2.3 PERFORMANCE METRICS CLASSIFICATIONS 24

2.3.1 GENERAL CATEGORIES OF PERFORMANCE METRICS 25 2.3.2 OPERATIONAL AND TACTICAL PERFORMANCE METRICS 25 2.3.3 LEADING AND LAGGING PERFORMANCE METRICS 26 2.3.4 DEVIL’S QUADRANGLE DIMENSIONS 26 2.3.5 ASSESSMENT OF PERFORMANCE METRICS CLASSIFICATIONS 26

2.4 PERFORMANCE MEASUREMENT APPROACHES 27

2.4.1 GOAL QUESTION METRIC 27 2.4.2 BALANCED SCORECARD 29 2.4.3 PROCESS-ORIENTED APPROACHES 31 2.4.4 OTHER APPROACHES 34

2.5 ASSESSMENT OF PERFORMANCE MEASUREMENT APPROACHES 35

2.6 CONCLUSIONS 36

3. PROCESS-ORIENTED E-PROCUREMENT PERFORMANCE MEASUREMENT (PEPM)

APPROACH 38

3.1 SHORTCOMINGS OF STUDIED PROCESS-ORIENTED APPROACHES 38

3.2 RATIONALIZATION OF REQUIRED STEPS IN PEPM APPROACH 39

3.3 OVERVIEW OF PEPM APPROACH AND UNDERLYING STEPS 41



3.4 STEP 1: E-PROCUREMENT CHAIN ANALYSIS 43

3.5 STEP 2: IDENTIFYING AND MEASURING E-SERVICES 44

3.6 STEP 3: IDENTIFYING SUPPORTING PROCESSES AND OPERATIONS 46

3.7 STEP 4: DEFINING PROCESS PERFORMANCE OBJECTIVES 48

3.8 STEP 5: PROCESS MODELING 49

3.9 STEP 6: DERIVING PERFORMANCE METRICS PER PROCESS 49

3.10 STEP 7: PERFORMANCE METRICS VERIFICATION AND VALIDATION 51

3.11 CONCLUSIONS 53

4. PEPM APPLICATION ON RABOSHOP 54

4.1. STEP 1 OF PEPM APPROACH 54

4.1.1 RABOSHOP AND E-PROCUREMENT 54 4.1.2 RABOSHOP CHAIN STRUCTURE 55

4.2 STEP 2 OF PEPM APPROACH 57

4.3 STEP 3 OF PEPM APPROACH 58

4.4 STEPS 4 – 7: PROCESS OBJECTIVES, MODELS, METRICS, AND VALIDATIONS 59

4.4.1 CATALOGUE BUYING PROCESS 60 4.4.2 FREE ORDER PROCESS 63 4.4.3 OPERATIONAL CONTRACT MANAGEMENT 65 4.4.4 REPLENISHMENT MANAGEMENT 66 4.4.5 CATALOGUE MANAGEMENT 68 4.4.6 PRODUCT CATEGORY MANAGEMENT 70 4.4.7 INCIDENT PROCESS 72 4.4.8 PROBLEM MANAGEMENT 75 4.4.9 CHANGE MANAGEMENT 77 4.4.10 RELEASE MANAGEMENT 80

5. ANALYSIS OF PEPM APPROACH 82

5.1 PEPM VERSUS THE STUDIED PERFORMANCE MEASUREMENT APPROACHES 82

5.1.1 PEPM APPROACH VERSUS ITIL 82 5.1.2 PEPM APPROACH VS. OTHER STUDIED PERFORMANCE MEASUREMENT APPROACHES 82

5.2 ANALYSIS OF STEPS OF PEPM APPROACH 84

5.2.1 PEPM E-SERVICES DISCOVERY AND MEASUREMENT 84 5.2.2 PEPM SUPPORTING PROCESSES IDENTIFICATION 84 5.2.3 PEPM PROCESS PERFORMANCE MEASUREMENT 85

6. CONCLUSIONS AND RECOMMENDATIONS 87

6.1 OPERATIONAL CONTROL IN E-PROCUREMENT CHAINS 87

6.2 PEPM APPROACH 87

6.3 PEPM CONTRIBUTIONS 88

6.4 RECOMMENDATIONS 90



6.5 FUTURE RESEARCH 90

6.6 DELIMITATION 91

7. REFERENCES 92

APPENDICES 96

APPENDIX A: BPMN 96

APPENDIX B: ITIL PROCESSES 96

APPENDIX C: ITIL PERFORMANCE INDICATORS 99

List of Figures

Figure 1: PEPM approach 3

Figure 2: Controlling e-services 4

Figure 3: Controlling supporting services 4

Figure 4: RCI Cost-Value-Risk principle (RCI Year Plan, 2009) 11

Figure 5: Research approach 15

Figure 6: Literature review approach 17

Figure 7: Procurement model (Van Weele, 2001; Harink, 2003) 18

Figure 8: Positioning of e-procurement tools in e-procurement process (Harink, 2003) 19

Figure 9: Three fundamental market models for e-marketplace (Meier and Stromer, 2009) 21

Figure 10: Devil's quadrangle dimensions (Rijers and Manson, 2005) 26

Figure 11: GQM hierarchical stucture (Basili et al., 1994) 28

Figure 12: Coordinates of Goal (Basili et al., 1994) 29

Figure 13: The four phases of the GQM method (van Solingen and Berghout, 1999) 29

Figure 14: Balanced Scorecard (Martinsons et al., 1999) 31

Figure 15: Concept of process model (Han & Kang, 2007) 32

Figure 16: ITIL processes 34

Figure 17: Top level objectives of PEPM approach 40

Figure 18: Controlling e-services of e-procurement system 45

Figure 19: performance metrics for QoS control 46

Figure 20: Controlling supporting operations of e-procurement system 47

Figure 21: PEPM approach 53

Figure 22: Closed loop: relation between contract, orders, and invoicing (RCI, 2009) 54

Figure 23: Involved parties and their relationships in Raboshop chain 56

Figure 24: To be controlled Raboshop supporting processes 58

Figure 25: second phase of PEPM approach 60

Figure 26: Constructs for operational control of e-procurement chains 88

Figure 27: Total overview of PEPM approach 89



List of Tables Table 1: E-procurement forms and explanations (De Boer et al., 2001) 20

Table 2: University of California performance metrics categorization 25

Table 3: ITIL V3 - Domains & Objectives 33

Table 4: PEPM's outputs of executing step 1 44

Table 5: PEPM process objective structure 48

Table 6: PEPM performance metric documentation 51

Table 7: UC performance metric quality criteria 51

Table 8: PEPM performance metric evaluation 52

Table 9: Raboshop underlying IT systems 55

Table 10: Raboshop core parties 55

Table 11: Raboshop interacting parties 56

Table 12: Raboshop e-services 57

Table 13: Raboshop critical supporting processes 59

Table 14: List of abbreviations 60



1. Introduction

This thesis is carried out in Rabobank Nederland. In the following section this organization

is further explained. Thereafter the thesis context, problem statement, the research

questions, research objective, and research approach are described.

1.1 Company profile

This section provides a brief explanation of the organization where the research was carried

out.

Rabobank Group

The research study is performed within Rabobank Nederland. Rabobank can be described as

an international financial service organization, executing ‚all finance services‛ in the

Netherlands. The independent local cooperative banks constitute the core business of the

Rabobank Group. Rabobank International, wholesale banking, and retail banking are the

other major constituents of the Rabobank Group.

This broad and leading financial group was founded in late nineteenth century as a small

collection of farm credit banks. The cooperative structure and local connectedness are

remained the main characteristics of Rabobank group in all these years.

The Rabobank Nederland is the central organization that coordinates major activities within

Rabobank Group departments, acting as a supportive and advisory role. In total, Rabobank

Group has more than nine million clients. There are one hundred and forty seven local

banks with more than one thousand branches.

Rabobank Concern Inkoop

Every organization needs necessary assortments and products to be able to operate their

regular activities. Depending on type of organization needed products may vary from simple

objects like pens and pencils to compound ones such as big machines in construction firms.

Rabobank Concern Inkoop (RCI) is the procurement department of Rabobank Nederland. It

mainly organizes the purchasing function of the Rabobank Group and aims to maximize the

procurement savings and to manage the risks by controlling the Group-wide purchasing

volume.

Every year, Rabobank Nederland purchases significant amount of goods and services. With

that, the purchasing function can be considered having a significant impact on the risk-



profile and financial results of the Rabobank Group. The principle of RCI is to maximize

value while minimizing costs and risks, see figure 4.

Value

Cost Risk

Figure 4: RCI Cost-Value-Risk principle (RCI Year Plan, 2009)

In order to facilitate the purchasing activities, RCI runs an electronic shop which is called

Raboshop. Raboshop is an electronic marketplace where suppliers offer their products

through product owners and exclusive clients (Rabobank Nederland, local banks, and

affiliated institutions) purchase the offered products.

Various products and services can be ordered through this electronic ordering system.

Posters, office products, business gifts, and IT products are examples of available items in

Raboshop assortment. The offered products must meet Rabobank standards and are being

monitored continuously in order to guarantee their quality.

1.2 Problem statement

Electronic procurement systems (e-procurement) that offer trading opportunities for both

suppliers and buyers are supported by complicated inter-organizational information

systems.

There is a chain of parties for supporting these types of large information systems.

Application management, functional management, client management, and suppliers are

examples of the involved parties.

In these complex systems, efficient business functioning and business continuity depend on

both individual operability of the participating parties and systems and inter-operability of

the chain participants.

Individual operability of the parties and systems is the primary requirement for the success

of any electronic organization. A low-performing supplier or a slow invoicing system can

hamper the operability of the total business drastically.



The collective efforts and smooth collaboration of the chain participants are the secondary

requirement for the business operability. A well-performing supplier cannot deliver the

products on time if the orders came too late due to a slow-performing ordering system.

Therefore the chain operability consists of individual and collective operability of the chain

participants.

The performance measurement is a key tool for efficient and effective control of such

complex situation. The fully functional points, weak areas, and potential problems can be

detected and addressed if the performance of the e-procurement chain can be measured in a

structured manner. At the same time, the performance measurement provides the chain

management with substantial information of the chain participants activities. The chain

management can use this information to give proper instructions to the stakeholders and to

enhance the performance level of the chain and consequently success of the e-procurement

business.

There are different studies that have been carried out to undermine the different

performance aspects of the e-procurement systems. Either they emphasize on the

performance of the system (e.g. Bremser and Chung, 2005) or they state the performance of

the individual party and organizational units (e.g. Gunasekaran et al., 2004). To the best of

author’s knowledge, the performance measurement of e-procurement chains is not

addressed in any scientific study.

Moreover these studies provide too many performance metrics if the chain management of

an e-procurement system wants to adapt and apply them for the performance measurement

of the chain.

Besides these, the provided performance metrics are too generic in most papers and do not

target a specific type of organization. Further, the methodology of applying the developed set

of performance metrics to a specific situation is not clarified.

Therefore, this thesis focuses on how the performance of an e-procurement chain can be

measured in an efficient and structural manner with the objective of providing the

management with the key insights of chain operability.

RCI, as the chain management of Raboshop, has high interests in measuring the

performance of underlying Raboshop chain. Through performance measurement of the

chain, RCI obtains insights on the operability aspects of the Raboshop chain.



1.3.1 Main research question

The main question for this research project is:



1.3.2 Sub research questions

In order to answer the main research question properly, it is split up into several sub

questions. Answering these questions will lead up to answering the main research question.

The following sub questions are formulated:

1. What is procurement and e-procurement?

The focus of the thesis falls on the e-procurement systems and their corresponding chains.

Therefore, the initial step for answering the main question is to study and understand the

basics of procurement and e-procurement. Based on the relevant literature, a brief

explanation and discussion of procurement and e-procurement concepts answers this

question.

2. What kind of e-procurement forms and underlying electronic market models are there?

Gaining knowledge of e-procurement types, underlying market models and their offered

services help to analyze the e-procurement systems and their corresponding chains.

3. What are the possible management levels? And at what level the chain management of e-

procurement system of study is operating on?

The levels of management define the scope of control activities. Understanding the

management levels and specifying scope of control are the objectives of this question.

4. What are the well-known approaches for performance measurement? And which performance

measurement approaches is the most suitable approach for the performance measurement of e-

procurement chains?

Acknowledging the existing basic performance measurement approaches and analyzing

their appropriateness for the performance measurement of e-procurement chains provide

the key prospects for performance measurement of e-procurement chains.

5. What are the important performance metrics categorizations? And which performance metrics

classifications are the most suitable for performance measurement of e-procurement chain?



The performance measurement of e-procurement chains is the main objective of this thesis.

Understanding the concepts of performance measurement and analysis of classifications of

performance metrics are the basics behind the performance measurement.

1.3.3 Research objective

This study aims at performance measurement e-procurement chains. This objective is

achieved by identifying important aspects of controlling and measuring of e-procurement

chains. All the findings are then processed and an exclusive performance measurement

approach for e-procurement chains is designed.

In addition to achieving performance objectives, applicability and efficiency factors are taken

into account while designing the e-procurement performance measurement approach. The

applicability refers to identifying and defining concrete steps on how the approach can be

implemented. Efficiency factor is about designing a fast-to-apply and effective application of

the method. Taking these aspects into account encourage the use and application of the

designed method among other existing performance measurement approaches.

1.3.4 Research approach

The answer to the main research question is composed of gained results from sub

questions.

In the first phase of the thesis (chapter 2) a literature review is carried out. In first section of

literature review, the basic concepts of procurement and e-procurement are studied.

Moreover, types of e-procurement systems are reviewed and ultimately elements of e-

procurement chain and its control levels are studied. The second section of literature study is

to gather knowledge about performance measurement concepts and existing performance

measurement approaches. These two steps provide the necessary knowledge to assess and

develop an exclusive approach for performance measurement of e-procurement chain.

The actual design and development of performance measurement approach is the second

phase of the approach (chapter 3). The steps of the approach and how these must be applied

are addressed.

The designed approach is evaluated on achieving performance objectives, applicability, and

fast-to-apply factors, in third phase of the research, by applying it to an industry case (chapter

4).

Finally, based on composition and application phases of the thesis the developed approach is

analyzed (chapter 5) and its strong and weak points are discussed.



Figure 5 illustrates the research approach steps.

Literature Review

Approach

Composition

Approach

Application

Approach Analysis

Chapter 2

Chapter 3

Chapter 4

Chapter 5

Figure 5: Research approach

1.3.5 Data collection methods

In the application phase of to be designed method, the required data are gathered by using

data collection methods. The following paragraphs elaborate the data collection methods in

more details.

Interviews

There are several forms of interviews according to Yin (2003). Structured and semi

structured forms are the most common applied types of interviews. In a structured one, each

interview is presented with either open-ended or close-ended question and the set of

questions are fixed. On the other hand, semi structured interviews are flexible allowing new

questions to be made as the interview progresses.

In the application phase of this research, the semi structured interviews are carried out to

gain the required knowledge and data.

Documentation archive

In order to assess the situation as well as to gather the required data for the application

phase of designed method available existing documentations of case study are used as

another form of data collection. Business plans, year plans, presentations, and other

organizational documents are the examples of such archival records that can be studied for

information gathering.



Participant observation

The project was carried out full-time at the research side itself and therefore participant

observation was applied as another source for data collection. Attending in periodic internal

meetings and informal discussions with people involved were the regular activities to gain

further knowledge of case study situation.



2. Literature review

In this chapter e-procurement concepts and performance measurement concepts are studied

and analyzed from different angles and perspectives. Figure 6 outlines steps taken towards

the completion of literature review.

E-procurement concepts

Performance measurement concepts

and metrics

Performance measurement

approaches

Processing the information

Approach Constructs

Section 2.1

Sections 2.2 - 2.3

Section 2.4

Figure 6: Literature review approach

The results of this literature study provide sufficient scientific knowledge for the creation of

performance measurement approach for operational control of e-procurement chains.

2.1 Electronic procurement (e-procurement) concepts

This section delves into the subject of procurement and provides insights e-procurement

domain.

The first two sub sections answer the first sub research question:

“What is procurement and e-procurement?”

2.1.1 Procurement

Procurement activities are an inseparable part of any organization. In larger organizations,

purchasing activities get complicated when various goods and services are needed for

different departments and sections.

Van Weele et al. (2003) define procurement as following: ‚Obtaining from external sources

all goods and services which are necessary for running, maintaining and managing the

company’s primary and support activities at the most favorable conditions‛

In general, procurement process is categorized into two sub processes (Harink, 2003):



- Procurement transaction process: It is about transaction-oriented procurement

- Procurement management process: It consists of activities for management of

procurement transaction process.

There are several steps defined within procurement transaction process. A procurement

process consists of six steps: specifying procurement strategy, selecting right suppliers,

contracting, ordering products and services, expediting and control of deliveries, and follow

up and evaluation (Van Weele, 2001). In addition, Harink (2003) adds procurement

management process. The Figure 7 illustrates the complete procurement process.

Procurement Management Process

Specifying Selecting Contracting OrderingExpediting

& Control

Follow up &

Evaluation

Figure 7: Procurement model (Van Weele, 2001; Harink, 2003)

2.1.2 E-procurement

Many organizations and firms began to use internet as a new way of doing business and this

triggered the rise of electronic commerce (e-commerce). According to Garrett and

Skevington (1999) e-commerce is trading by means of new communication technology (e.g.

internet). It includes all aspects of trading, including commercial market making, ordering,

supply chain management, and the transfer of money.

Business efficiency, increased automation of processes, retained and expanded customer

base, and reduced information costs are some of the major benefits of e-commerce (Kuzic et

al., 2002).

Through the rise of e-commerce, various products and services are available online and

organizations are able to procure their needed products and services online. This initiative

was the start of new era for electronic procurement (e-procurement).

E-procurement refers to all of the connective processes between companies and suppliers

that are enabled by electronic communication networks (Meier and Stormer, 2009). Harink

(2003) defines e-procurement as using internet technology for procurement process. Thus

the ultimate objective of e-procurement systems is to provide electronic services (e-services)

that facilitate procurement process.



By automating processes and workflows associated with purchasing, the firm expects to

increase the productivity of its purchasing agents, lower purchase prices of different types of

goods and services, streamline the information flow, business processes, and workflows

involved in purchasing, eliminate maverick buying (i.e., buying from unauthorized vendors),

reduce order fulfillment and processing times, reduce the number of suppliers the firm is

dealing with, streamline invoice reconciliation and dispute resolution, reduce the

administrative processing cost per purchase order, integrate budgetary controls into the

procurement process, minimize human errors in the buying and shipping processes, and

monitoring and regulating buying behavior (Turban et al., 2005).

From a different perspective, e-procurement can be considered as a way to benefit from real-

time information sharing and it provides the opportunity to take advantage of total quality

management and supply chain management (Sriram and Stump, 2004).

2.1.3 E-procurement forms

This section delves into the forms of e-procurement and answers the second sub research

question:

‚What kind of e-procurement forms and underlying electronic market models are there?”

Harink (2003) identifies e-procurement forms for automating different steps in

procurement process. E-sourcing, e-tendering, e-reverse auctioning, e-contract management,

e-ordering, and web-oriented ERP are the identified forms of e-procurement. In figure 8 the

positioning of these e-procurement forms are plotted on the procurement process.

Specifying Selecting Contracting Ordering Control Follow up

Purchasing Intelligence

E-sourcing

E-tendering

E-reverse auctioning

E-contracting

Web-based ERP

E-ordering

Network (E-marketplace)

Figure 8: Positioning of e-procurement tools in e-procurement process (Harink, 2003)

The brief explanations of the e-procurement forms are provided in table 1:



E-procurement forms Description

E-ordering

The process of creating and approving purchasing requisitions, placing purchase orders and receiving goods and services by using a software system based on internet technology. The goods and services ordered are MRO (Maintenance, Repair and Operation) or NPR (Non-Product Related). The software systems contains of an ordering catalogue system preferably used by all employees of an organization.

Web-based ERP The same process as in the case of e-MRO, but with web-oriented ERP, the goods and services ordered are product related.

E-sourcing

Includes the process of identifying new suppliers for a specific category of purchasing requirements using internet technology. It can lead to identification of new suppliers, which can lead to increased competitiveness in the tendering process. it is also a way of decreasing the supply risk associated with a purchasing category.

E-tendering

The process of sending requests for information and prices to suppliers and receiving the responses of participants using internet technology. It sometimes also concerns the analysis and comparison of responses, but excludes the process of closing the deal.

E-reverse auctioning

An auction enables a supplier to sell goods and services to a number of known and unknown buying organizations. Within a relatively short time frame, the buying organization involved submits bids for the goods and services that are auctioned. A reverse auction is the opposite and enables one purchaser to buy goods or services from a number of known or unknown suppliers.

E-contracting The use of a contract management system linked to an automated ordering and /or payment system.

Purchasing Intelligence Concerns the process of uploading purchasing data and the analysis of this data into valuable purchasing information.

Table 1: E-procurement forms and explanations (De Boer et al., 2001)

2.1.4 E-marketplaces

The place where the e-procurement forms are offered is referred to as e-marketplaces (i.e.

electronic marketplace) (Harink, 2003).



These e-marketplaces are spots on the internet where buyers and sellers can do business

with each other. There are various different types of marketplaces. One can distinguish

between buy-side marketplaces which are set by the buyer, sell-side market places set up by

sellers and neutral marketplaces designed for both buyer and seller (Van Weele et al., 2003).

Figure 9 illustrates the different types of e-marketplaces along with examples from industry

for each type of e-marketplaces.

Figure 9: Three fundamental market models for e-marketplace (Meier and Stromer, 2009)

In the sell-side market model, the supplier provides the purchase software and an electronic

catalogue. Here the buyer must register with each supplier and familiarize himself with

different software solutions and navigational aids. Some suppliers with sell-side solutions

provide extensive functions for personalization, for production configuration, or for

compatibility testing.

In the buy-side market model, the buyer must run and maintain the appropriate software

together with extracts from the product catalogue. Use within a company increases, since the

procurement can be realized with a uniform product view.

In the (neutral) market model the required software solution used simultaneously by several

companies (buyers) as well as by several suppliers (sellers). The third-party can uniformly

display and describe the products with his software solution.



Based on the above definitions, the services that the e-procurement systems with different

underlying e-marketplaces model provide may vary. This implies taking the model of e-

marketplace into account while analyzing the e-procurement systems and their offered

services.

2.1.5 E-procurement chain and management

This section explains the possible control levels of e-procurement chain that the

management is facing it. The descriptions are rather brief because delving into the matter

might cause deviation of the thesis’s objective. The brief explanations are necessary to clarify

the level of control (i.e. management) on e-procurement chain. The third sub research

question is answered:

“What are the possible management levels? And at what level the chain management of e-

procurement system of study is operating on?”

Positioning the chain management of e-procurement business is a governance issue,

depends on the nature of the business and might differ within every e-procurement

business. Therefore the focus here is on specifying control level (i.e. management level) of

the e-procurement chain for this study.

Primarily, the e-procurement business is enabled by its underlying (inter-organizational)

information systems. These information systems enable different forms of e-procurement

forms and underlying e-marketplaces.

Secondly, based on the e-procurement forms and underlying e-marketplace types, the e-

procurement system is used by buyers and providers (suppliers) of the products and

services. In addition, a group is responsible for developing and maintaining of the

underlying information systems for the enablement of the e-procurement system.

Based on above points, the chain of an e-procurement business consists of parties and

information systems. Hence, controlling chain implies controlling parties, information

systems and their interactions.

The subsequent point is to understand the level of management. Anthony (1965) defines

three levels of management activities:

1. Operational control, which is ‚the process of assuring that specific tasks are

carried out effectively and efficiently.‛

2. Management control, which is ‚the process by which managers assure that

resources are obtained and used effectively and efficiently in the accomplishment

of the organization’s objectives‛.



3. Strategic planning, which is ‚the process of deciding on the objectives of the

organization, on changes in these objectives, on the resources used to obtain

these objectives, and on the policies that are to govern the acquisition, use and

disposition of these resources.‛

The thesis’s objective is to discover an efficient method for performance measurement of an

e-procurement chain to provide supportive information of the chain operability for its

management. Thus the operational control is the appropriate level for performance

measurement’s focus of this study.

Therefore, the term chain management, within the context of this study, focuses on

operational control level of e-procurement chains.

2.2 Performance measurement concept

Prior to any analysis and application of performance measurement methods, it is necessary

to understand the performance measurement definition and its advantages.

There are several definitions of performance measurement. The U.S. General Accounting

Office (GAO, 2005) provides a comprehensive definition of performance measurement:

‚Performance measurement is the ongoing monitoring and reporting of program accomplishments,

particularly progress towards pre-established goals. It is typically conducted by program or agency

management. Performance measures may address the type or level of program activities conducted

(process), the direct products and services delivered by a program (outputs), and/or the results of

those products and services (outcomes). A program may be any activity, project, function, or policy

that has an identifiable purpose or set of objectives.”

Performance measures quantitatively tell us something important about our products,

services, and the processes that produce them. They are a tool to help us understand,

manage, and improve what our organizations do. Effective performance measures can let us

know:

- How well we are doing.

- If we are meeting our goals,

- If our customers are satisfied,

- If our processes are in statistical control, and

- If and where improvements are necessary

They provide us with the information necessary to make intelligent decisions about what we

do.



A performance measure is composed of a number and a unit of measure. The number gives

us a magnitude (how much) and the unit gives the number a meaning (what). Performance

measures are always tied to a goal or an objective (the target). Performance measures can be

represented by single-dimensional units like Days, meters, nanoseconds, dollars, number of

reports, number of errors, number of CPR-certified employees, and length of time to design

hardware. They can show the variation in a process or deviation from design specifications

(Stroh and Artley, 2001).

Mark Graham Brown, in his book (Brown, 2010), mentions three important benefits about

why performance measurement has to be used:

- Measurement provides concrete data on which to make sound business decisions,

thus reducing the urge to manage by intuition.

- Measurement identifies areas needing attention and enables positive influence in

that area.

- Improvement is impossible without measurement. If you do not know where you are,

then you cannot know where you are going and you certainly cannot get to where you

want to be.

In a different point of view, performance measurement is necessary when managing

different stakeholders. Performance measurement is fundamental to organizational

improvement. The importance of performance measurement has increased with the

realization that to be successful in the long-term requires meeting (and therefore measuring

performance against) all stakeholders' needs including customers, consumers, employees,

suppliers, local community stakeholders, and shareholders. While the importance of

performance measurement is difficult to quantify it is evident that in virtually all texts,

research, and case studies on organizational improvement, that performance measurement

plays a central role. It is worth noting that performance measurement is a requirement for

benchmarking and business excellence (BPIR).

2.3 Performance metrics classifications

The concept of performance measurement was explained in the previous section. This

section delves into the basic outputs of any performance measurement approach, the so

called performance metrics. This part of literature study will answer the fifth sub research

question:

“What are the important performance metrics categorizations? And which performance metrics

classifications are the most suitable for performance measurement of e-procurement chain?”



There are several classifications of performance metrics. These classifications divide

performance metrics into several categories. The following sub sections explain the most

well known classifications of performance metrics.

2.3.1 General categories of performance metrics

University of California defines five general categories for performance measures (Stroh and

Artley, 2001). Table 2 explains these categories.

Efficiency A process characteristic indicating the degree to which the process

produces the required output at minimum resource cost. (Are we doing

things right?)

Quality The degree to which a product or service meets customer requirements

and expectations.

Timeliness Measures whether a unit of work was done correctly and on time.

Criteria must be established to define what constitutes timeliness for a

given unit of work. The criterion is usually based on customer

requirements.

Productivity The value added by the process divided by the value of the labor and

capital consumed.

Safety Measures the overall health of the organization and the working

environment of its employees.

Table 2: University of California performance metrics categorization

They mention that depending on the type of organization and the performance targets, this

set of categories can be adapted to meet their situation.

2.3.2 Operational and tactical performance metrics

From a different perspective, performance metrics can be partitioned into operational,

tactical and strategic categories (Reijers, 2002). Operational performance metrics are

covering the measures for day-to-day activities meanwhile tactical and strategic metrics are

about the long term targets that the organization is perusing.

There are different types of performance metrics for different types of users. Senior

managers need in some high level performance metrics that are supportive for decision

making while operational managers need low level performance metrics that provides them

with enough insights about the day-to-day activities and operations. Therefore it is necessary



to distinguish different levels of performance metrics to address the needs of different types

of employees within an organization.

2.3.3 Leading and lagging performance metrics

This classification of performance metrics is time-oriented. A leading performance metric is

suggestive of future performance. The benefit of leading metrics is that they enable

managers to proactive action to help shape future performance (Stroh and Artley, 2001).

Average handle time, number of contacts and average speed of answer are some examples of

leading performance metrics.

Lagging performance metrics focus on results at the end of a time period. They normally

characterize the historical performance. Total number of problems and total number of

incidents are some example of the lagging indicators. These indicators are used

operationally on daily or periodic basis so that performance can be diagnosed and reviewed.

2.3.4 Devil’s quadrangle dimensions

The devil’s quadrangle distinguishes four main dimensions when evaluating the redesign of

business processes (Reijers and Manson, 2005). Time, quality, cost, and flexibility are the

four dimensions. Figure 10 illustrates the four dimensions and their relationships. The

special characteristic of devil’s quadrangle classification is that it focuses on the process

performance.

Figure 10: Devil's quadrangle dimensions (Rijers and Manson, 2005)

2.3.5 Assessment of performance metrics classifications

The performance metrics are classified according to the domain where they are applied to.

The general categories of UC define basic types that a performance metrics can be assigned



to. This classification is independent of where and for which objectives the performance

metrics are used.

The operational and tactical classification of performance metrics does not look at the type of

performance metric from the performance aspect viewpoint, but they classify the

performance metrics based on managerial levels. This way of classification is useful when

the management at each level wants to create its own set of performance metrics. The

problem with this classification is the how question. Determining a performance metrics

belongs to which level is not straight forward.

The leading and lagging categorization of performance metrics is useful when the

management either mainly emphasizes on the historical performance of the business

(lagging performance metrics) or is interested in predicting of the business performs in the

future. This type of categorization is useful to draw decisions based on the historical

performance and prediction of the future performance, especially in highly competitive

markets.

Devil’s quadrangle dimensions are usually devoted to process performance measurement

(Reijers and Manson, 2005). These dimensions are suitable especially in case of business

process reengineering. The improvements of process reengineering can be measured

against these performance dimensions.

2.4 Performance measurement approaches

There are several well-known widely recognized approaches that provide methods and tools

for the task of performance measurement. By reviewing the approaches in this section the

fourth sub research question is answered:

“What are the well-known approaches for performance measurement? And which performance

measurement approach is the most suitable approach for the performance measurement of e-

procurement chains?”

2.4.1 Goal Question Metric

The Goal Question Metric (GQM) approach is based on the assumption that for an

organization to measure in a purposeful way it must first specify the goals for itself and its

projects, then it must trace those goals to the data that are intended to define those goals

operationally, and finally provide a framework for interpreting the data with respect to the

stated goals (Basili, et al., 1994).

The GQM model is a hierarchical structure (see figure 11) that has three levels:



1. Conceptual level (Goal): A goal is identified for an object, for a variety of reasons, with

respect to various models of quality, from various points of view, relative to a

particular environment. Objects of measurement are products, processes, and

resources.

2. Operational level (Question): A set of questions is used to characterize the way the

assessment/achievement of a specific goal is going to be performed based on some

characterizing model. Questions try to characterize the object of measurement with

respect to a selected quality issue and to determine its quality from the selected

viewpoint.

3. A set of data is associated with every question in order to answer it in a quantitative

way.

Figure 11: GQM hierarchical stucture (Basili et al., 1994)

Basili et al. (1994) state that the process of setting the goal is critical to the successful

application of the GQM approach and that is why they support it by specific methodological

steps.

Each goal has four dimensions: issue, viewpoints, objects (products, processes, resources),

and purpose (see figure 12). Issue and purpose of the goal can be derived by analyzing

corporate policy statements, strategic plans, and interviews of the relevant subjects in the

organization. Object coordinate of the Goal is derived by specifying process and product

models. The model of the organization provides the viewpoint coordinate of the goal.



Figure 12: Coordinates of Goal (Basili et al., 1994)

In a further development of this method, van Solingen and Berghout (1999), introduce a

four phases approach for the implementation of the GQM method (see figure 13). Planning,

definition, data collection and interpretation are these phases. In the planning phase a

project for measurement application is chosen, defined, characterized and planned,

resulting in a project plan. During the definition phase goals, questions, and metrics are

defined and documented with the help of interviews. During data collection phase the actual

data collection takes place, resulting in collected data. In interpretation phase, the collected

data is processed with respect to the defined metrics into measurement results that provide

answers to the defined questions, after which goal attainment can be evaluated.

Figure 13: The four phases of the GQM method (van Solingen and Berghout, 1999)

The outstanding aspect of GQM method is being a goal-oriented approach. Mapping goals

with metrics validates the correctness of this approach.

2.4.2 Balanced scorecard

Perhaps the most well-known of the performance approaches is balanced scorecard (BSC)

framework due to its extensiveness and comprehensiveness. Kaplan and Norton introduced

a balanced framework that covers both financial and nonfinancial performance measures for



the organizations (Kaplan and Norton, 1992). This framework covers different aspects of

organization’s performance by defining several performance perspectives.

BSC framework for performance measurement was introduced by Kaplan and Norton

(Kaplan and Norton, 1992). This framework defines performance perspectives and provides

guidelines for defining performance metrics for each of these perspectives. Financial,

customer, internal business, and innovation and learning perspectives are the four

perspectives introduced in the framework. Ever since, many organizations and firms have

applied this method to their situations. These four perspectives measure the business or an

organization from different angles. In the following paragraphs a short explanation of each

perspective has been provided.

Financial perspective: These measures reflect the financial performance of the business.

Return on investment (ROI) and cash flow are some of the examples of these measures. This

perspective answers on how we look to shareholders.

Customer perspective: This perspective is concerned with the direct impacts that the measures

will have on the customers. Customer satisfaction level and number of complaints are some

of these metrics. The organization will discover how the customers see the business by

applying this perspective.

Internal business perspective: Performance metrics in this perspective answer the question on

at what we must excel at. The business finds out where it performs effectively and efficiently.

Average processing time and number of incidents are typical examples of metrics in this

perspective.

Innovation and learning perspective: It encourages the identification of measures that answer

the question whether we can continue to improve and create value.

All of these explanations and their possible relations are outlined in figure 14.

The proposed balanced scorecard approach provides a clear roadmap about the performance

measures and the ultimate target groups in any organization. The framework emphasizes

that financial measures do not reflect the total picture of business performance and

therefore they have defined non-financial perspectives to their framework. On this way the

introduced framework balances between different perspectives when measuring the

performance of the business.



Figure 14: Balanced Scorecard (Martinsons et al., 1999)

Further, Kaplan and Norton (1993) introduced a brief description of a process that enabled

managers to design balanced measurement systems. Neely et al. (2000) state that as the

balanced scorecard has grown in popularity, most of relative performance measurement

design processes have ended up as rather open-ended and vague statements, including the

process introduced by Kaplan and Norton (1993).

2.4.3 Process-oriented approaches

Prior to introducing the process-oriented approaches, a definition of a business process is

necessary. There are several business process definitions and Ko (2009) discusses these and

provides this definition of the process: “A business process is a series or network of value-added

activities, performed by their relevant roles or collaborators, to purposefully achieve the common

business goal”.

Process-oriented performance measurement approaches focus on measuring the

organization’s business processes and ignoring the organizational structures. On this way,

an enterprise is seen as a collection of core business processes that span functional

boundaries and are not as a set of discrete functions and business units. Figure 15 illustrates

the process-oriented concept for organizational analysis.



Figure 15: Concept of process model (Han & Kang, 2007)

In the following two studied process-oriented performance measurement approaches are

explained.

Kueng (2000) approach:

This approach provides a process performance measurement system (PPMS) for measuring

the performance of a process. The main objective of the PPMS is to provide comprehensive

and timely information on the performance of business processes. This information can be

used to communicate goals and current performance of a business process directly to the

process team, to improve resource allocation and process output regarding quantity and

quality, to give early warning signals, to make a diagnosis of the weaknesses of a business

process, to decide whether corrective sections are needed to assess the impact of actions

taken. Further Kueng defines innovation aspects, social aspects, customer aspects, employee

aspects, and financial aspects as the performance aspects in his methodology.

PPMS approach consists of the following steps: (1) eliciting appropriate process performance

indicators; (2) determining target values for each indicator; (3) developing methods and

instruments to gather the data; and (4) creation an information system that stores collected

data, distributes results and provides easy access to various user categories.

To identify the performance metrics, he prefers to elicit performance metrics from scratch

rather than using an existing generic set of performance metrics. So the first step of the

PPMS consists of the following iterative sub steps: (1) Define high-level process goals; (2)

Derive performance indicators; (3) Derive sub goals; (4) Refine and modify goals.



Kueng concludes that the process performance measurement is a necessity for the modern

process-oriented organizations that change from time to time .

ITIL:

Information Technology Infrastructure Library (ITIL) is a well-known standard for IT

governance domains. It introduces set of processes for managing an IT-organization and

provides a set of performance metrics for each of the processes. It is a process-oriented

performance measurement industry standard that is very applicable for many kinds of

organizations. ITIL gives detailed descriptions of a number of important IT practices and

provides comprehensive checklists, tasks and procedures that any IT organization can tailor

to its needs. ITIL provides several supportive processes that organizations may apply those to

their situations. Through this standard an organization can be represented with a set of

processes that cover many aspects of the organization.

ITIL version 3 encompasses multiple domains with multiple processes belonging to each of

these domains. Table 3 explains each of the defined domains.

ITIL domain Domain Objective

Service Strategy

To provide guidance on how to design, develop and implement Service Management. It is about ensuring that IT organizations are in position to achieve operational effectiveness and to offer distinctive services to their customers. Its ultimate goal is to make the IT organization think and act in a strategic manner.

Service Design To design and develop IT services. Its scope includes the design of new services, as well as changes and improvements to existing ones.

Service Transition To build and deploy IT services. It also makes sure that changes to services and Service Management processes are carried out in a coordinated way

Service Operation

To make sure that IT services are delivered effectively and efficiently. This includes fulfilling user requests, resolving service failures, fixing problems, as well as carrying out routine operational tasks.

Continual Service Improvement

The Continual Service Improvement process uses methods from quality management in order to learn from past successes and failures. It implements a closed-loop feedback system as specified in ISO 20000 as a means to continually improve the effectiveness and efficiency of IT services and processes.

Table 3: ITIL V3 - Domains & Objectives



Figure 16 illustrates the belonging ITIL processes for each ITIL domain:

Continual Service

Improvement

Service Operation

Service Transition

Service Strategy

Service Design

- Service Portfolio

Management

- Financial Management

- Service Catalogue

Management

- Service Level Management

- Risk Management

- Availability Management

- IT Service Continuity

Management

- IT Security Management

- Compliance Management

- IT Architecture Management

- Supplier Management

- Change Management

- Project Management

- Release and Deployment

Management

- Service validation and

testing

- Application developement

- Configuration Management

- Knowledge Management- Event Management

- Incident Management

- Problem Management

- IT Operations

Management

- Service Evaluation

- Process Evaluation

- Definition of Improvement

Initiatives

- CSI Monitoring

Figure 16: ITIL processes

Besides defining set of processes and procedures, ITIL provides key performance indicators

(KPI) per process as well. These performance indicators can be used as a starting point for

performance measurement of each process in e-procurement environment.

The ITIL process objectives and the documented KPI are attached in appendix B and

appendix C respectively.

2.4.4 Other approaches

Traditional methods mainly focus on the financial aspects of performance. DuPont and

Activity Based Costing are methods that mainly pay attention to the financial performance of

the organization. On the contrary methods like Self-Assessment and competitive

benchmarking are devoted to non-financial performance of the organizations (Kueng and

Krahn, 1999). In the latter approach, companies measure their performances by using the

set of award criteria provided by the Malcolm Baldridge National Quality Award (MBNQA)

from the United States or European Foundation for Quality Management (EFQM)

frameworks. The benefits using this self-assessment framework is that it produces an



objective identification of current strengths and areas for improvement and provides a useful

analysis of an organization’s capability, which is of real interest to potential customers.

2.5 Assessment of performance measurement approaches

This section considers the applicability of the performance measurement approaches

studied in section 3.3 to the thesis’s objective, i.e. the performance measurement approach

for e-procurement chain.

The GQM measurement approach is a goal-oriented top-down approach that can be applied

at all levels of the organization. This approach targets development of performance goals for

products, processes, and resources. The focus of this study is to measure performance of e-

procurement chains. Application of GQM method can be useful to define the goals and sub

goals in a systematic manner but further this method relies on producing appropriate

questions that answering them leads to defining performance metrics for the appointed

goals. To do so, designing questions heavily depends on the knowledge and experience of

organization’s experts. Further more, measuring all kinds of objects within e-procurement

chain can get extensive and may lead to infeasible amount of performance data.

In addition, the implementation of GQM, in case of a complex situation, can be very time-

consuming since the questions are needed to be developed and thereafter performance

metrics are drawn upon for each identified question.

From implementer point of view, development of questions requires extensive knowledge of

the domain under analysis and performance measurement experience.

Hence GQM method is hardly suitable for a fast-to-apply and efficient performance

measurement of e-procurement chains.

The BSC framework measures the performance of an organization from different

perspectives. Although the approach is easy to understand and measurement focused, BSC

performs at a high level of corporation and consequently is more suitable for organizational

development at a business management level. The BSC’s high-level deployment makes it

less suitable for implementation in operational control level. Moreover, the framework pays

less attention on how the performance measures are gathered. As stated by Neely et al.

(2000), the implementation of BSC is rather vague. Therefore the BSC approach is less

suitable for a fast-to-apply and efficient performance measurement of an e-procurement

chain.

The aspect of being process-oriented makes the application of an approach independent of

organizational structures. This aspect is very important for the modern organizations that



change from time to time. Although the actors and resources of the processes may change

due to organizational changes, but the essence of processes remain the same.

The studied process-oriented approaches directly state that in order to gain control of the

situation, the processes must be controlled and managed. Kueng’s approach focuses on how

a single process can be measured and ITIL comes up with set of processes to control and

manage for an IT situation.

E-procurement chain composed of enabling IT systems and parties. In modern day e-

procurement systems, these systems and parties can be very complex and very wide-spread.

Through process-oriented approach the e-procurement chain can be defined as set of

processes and hence the complexity of structure of systems and interacting parties can be

avoided. Gaining control on processes and measuring their performance informs the chain

management of the level of operability of the chain. Hence the process-oriented approaches

are the most suitable approaches towards achieving the thesis’s objective. Kueng approach is

appropriate because it provides the method to measure a process. The fact that e-

procurement system and its services are primarily of IT type, the set of ITIL processes can be

used when applying a process-oriented approach for e-procurement chains.

By focusing on ongoing processes within the belonging chain, the management can monitor

these processes, detect the problems and bottlenecks, and take responsive decisions upon.

Among the other approaches mentioned in sub section 3.3.5, DuPont and Activity-oriented

costing are more suited for measurement of financial performance of the organizations. On

the contrary, the performance measurement of e-procurement chain focuses on the

operability of the system and its chain rather than financial performance of the whole

business. The self-assessment method is a generic industry approach for performance and

quality measurement and is not appropriate for the specific target of this study.

Concluding to this assessment section, the process-oriented approach is the underlying

approach selected for the development of a performance measurement approach for e-

procurement chains.

2.6 Conclusions

The objective of literature review phase of the thesis was to accumulate the necessary

knowledge for the construction of a performance measurement approach for operational

control of e-procurement chains.

The concepts of procurement and the classifications of e-procurement forms were studied. It

was understood that each e-procurement systems can be launched on different types of e-



marketplace models. Buyer-side, seller-side, and the neutral models were the most well-

known types of e-marketplaces.

Further the typical structure of e-procurement chains and the possible management level of

the chain were reviewed. Based on the thesis’s objective, the operational control was

assigned as the level of management for performance measurement of e-procurement

chains.

The next step in the literature review chapter was to gain insights into performance

measurement concepts and some of the well-known approaches for performance

measurement.

The assessment on the studied performance measurement approaches resulted in selecting

process-oriented performance approach as the underlying approach for the development of a

performance measurement approach in this study.

Based on the gained knowledge in the literature review phase, in the next chapter an

approach for the performance measurement of e-procurement chains is developed and

explained.



3. Process-oriented E-procurement Performance Measurement

(PEPM) Approach

Based on the findings of the literature review and taking the thesis’s objective into account,

this chapter introduces a process-oriented performance measurement approach for the

operational control of e-procurement chains (i.e. PEPM approach).

Prior to the introduction of PEPM approach, an analysis of the studied process-oriented

performance measurement approaches is carried out to assess the need for the development

of a new approach. Section 3.1 states shortcomings and deficiencies of the studied process-

oriented approaches. Thereafter, in section 3.2, the required steps in PEPM approach are

rationalized. Subsequently, sections 3.3 to 3.10 introduce and explain the designed approach.

3.1 Shortcomings of studied process-oriented approaches

This section assesses the deficiencies of the studied process-oriented approaches. The

arguments rationalize the need for the introduction of a specific process-oriented

performance measurement approach for operational control of e-procurement chains.

Shortcomings of Kueng (2000) approach

The Kueng (2000) approach only explains the necessary steps of how a business process is

measured in a structural way but does not address how the process itself is identified. The

method does not provide any solution for the process identification. Application of Kueng

approach for process-oriented performance measurement of e-procurement chain requires a

process as input. Therefore prior to application of Kueng approach, there should be a

method to generate processes that needs to be controlled and measured.

In addition, in Kueng approach, a stakeholder-approach is introduced for defining

performance aspects and deriving performance measures. The main emphasis of the

approach is on deriving performance metrics from the scratch instead of using from a

standard existing set of performance metrics. Eliciting the performance metrics from scratch

can be very time-consuming (in case there are many processes to be measured) and requires

substantial knowledge of performance measurement to implement it.

Shortcomings of ITIL

ITIL approach provides a large set of management processes and defines performance

metrics for each of these processes. The main objectives of ITIL are to align IT services with

the current and future needs of the business and its customers, to improve the quality of IT

services, and to reduce the long-term cost of service. Although ITIL provides useful set of



management processes and provides performance indicators, but the main objective of ITIL

is not performance measurement. Therefore, only application of ITIL to measure

performance of e-procurement chains is not complete. On the contrary to Kueng approach,

ITIL does not provide a method to measure a process, but provides set of processes that

require controlling and measuring.

Furthermore, controlling and measuring all of ITIL processes is an infeasible task for the

chain management of an e-procurement system. Too many processes can lead to an

overwhelming number of measures and resulting data.

In addition, although the process performance metrics can be used as an initial reference for

the identification of the performance metrics for a specific case, the identified process

performance measures do not address what type of process performance aspects and which

part of the process are targeted.

Conclusions

These deficiencies in the studied process-oriented approaches were the reasons towards

developing a more complete process-oriented approach that targets the performance

measurement in e-procurement chains and removes the deficiencies of these process-

oriented approaches. But prior to introducing and explaining the PEPM approach, it is

necessary to clarify and rationalize which steps are required for the new approach. The

following section delves into this matter.

3.2 Rationalization of required steps in PEPM approach

This section argues, rationalizes and introduces the required steps to be developed for the

PEPM approach.

According to the assessment of performance measurement approaches in section 2.5, the

process-oriented approach was found the most suitable approach for the operational control

of e-procurement chains. The main idea behind this approach is to see the whole business as

set of processes (see section 2.4.3). Therefore, in order to know what to control and measure

in e-procurement system chain, based on the process-oriented approach, the first step is to

define e-procurement system and its chain as set of processes. Consequently the identified

processes can be measured. In the following paragraphs the set of processes that needs to be

controlled and measured are discussed and rationalized.

According to Dumas et al. (2001) and Jansen-Vullers and Reijers (2003), the electronic

services (e-services) offered by business to business e-commerce (and in particular in e-

procurement systems) are either simple activities or complex tasks ((semi) automated

business processes). On the other hand, according to the definitions of the e-procurement

provided in section 2.1.2, the main contribution of e-procurement systems to the



procurement process is offering e-services that automate some steps of procurement

process. The type and nature of the offered e-services differ depending on the e-procurement

forms and the underlying e-marketplace model. Due to electronic nature of offered services,

any interruption in delivering these e-services results in a huge impact on the operability of

the e-procurement chain. Hence monitoring and controlling the operation of offered

services is essential for the operability level of the e-procurement chain.

The fact that the offered e-services within the e-procurement context are primarily of process

type implies that first step in PEPM approach is to control and measure these e-services.

In addition, ITIL provides set of supporting processes for enabling and maintaining the

offered e-services (divided in five specific domains, section 2.4.3). ITIL emphasizes that

these supporting processes should be managed to assure the smooth management of IT

services (i.e. e-services). Therefore along side to controlling e-services, the set of supporting

processes has to be controlled and measured.

Thus the operational control of e-procurement chain can be ensured with a process-oriented

approach through controlling and measuring both offered e-services and the set of

supporting processes. Figure 17 illustrates this.


Operational Control of E-procurement chain

Controlling E-procurement services

Figure 17: Top level objectives of PEPM approach

Controlling e-procurement services and supporting operations is done, according to the

thesis’s objective, through performance measurement. Therefore the subsequent stage in

PEPM approach is to explain how all these services and supporting processes can be

measured.

Now that main phases of PEPM approach are specified, the following sections introduce the

approach in detail.



3.3 Overview of PEPM approach and underlying steps

The objective of the PEPM approach is to produce and identify performance metrics for the

operational control of e-procurement chain in a fast-to-apply and efficient manner.

The development of each step is supported by the use of relevant solutions of the studied

approaches and industry standards.

In this section an overview of the steps of the approach is provided where each step is

detailed with (1) Rationale: why this step, (2) Purpose: objective of the step, (3) Procedure:

how the step should be executed, and (4) Output: what are the expected results after

application of the step.

1. Identify e-procurement type and analyze the chain structure:

a. Rationale: There are different e-procurement forms, and the structure of the

belonging chain differs depending on e-marketplace model. Therefore the first

step towards process performance measurement is to identify e-procurement

form, its underlying e-marketplace business model, chain participants, and the

underlying enabling systems.

b. Purpose: This leads to the recognition of the e-procurement types and chain

participants.

c. Procedure: Use data collection methods introduced in section 1.3.5.

d. Output: The e-procurement types and its underlying e-marketplace business

model, structure of the e-procurement chain.

2. Discovering and measuring e-procurement e-services:

a. Rationale: Based on the arguments in section 3.2, in order to achieve operational

control on e-procurement system, the first step is to identify the set of offered e-

services. Thereafter their performance must be measured. Measuring the e-

services provides substantial information of how well the e-services are operating

and what are the quality levels of the e-services.

b. Purpose: To identify the set of offered e-services, define performance aspects for

e-services and produce performance metrics for each of e-services.

c. Procedure: Analyze e-procurement system and identify its e-services (data

collection method), Use performance aspects for e-services and identify

corresponding performance metrics.



d. Output: Set of e-services and performance metrics are produced in this step.

3. Identify supporting operations

a. Rationale: According to reasons stated in section 3.2, supporting processes of e-

procurement system should be identified and each process needs to be measured.

The measurement produces metrics to obtain insights of how well the supporting

processes are operating.

b. Purpose: To identify the set of to be controlled and measured supporting

processes for the operational control of e-procurement chain

c. Procedure: Use existing set of ITIL supporting processes, apply selection criteria.

d. Output: Recognition of set of to be monitored supporting processes.

4. Determine performance objectives per process:

a. Rationale: The first step for the process performance measurement is to

undermine why we are measuring (Kueng, 2000). A lot of measurement points

can be set while measuring a process and in doing so, drawing conclusions on the

performance of the process gets aimless if there are no process performance

objectives. Therefore to get the optimal number of performance metrics that

assess the performance of the process, process performance objectives should be

defined.

b. Purpose: To ensure there is a clear purpose for producing each process

performance metric.

c. Procedure: Enquire chain management with the help of the PEPM composed

process performance aspects.

d. Output: List of process performance objectives for each process.

5. Model processes:

a. Rationale: Modeling the process has several benefits to process performance

measurement. According to Brockers et al. (1996) the following are the major

benefits for process modeling in measurement programs: (1) Information about

the objects to be measured: before an attribute of a performance can be

measured, it is necessary to identify the corresponding product, resource or

process precisely. (2) Information about the resource who measures: process

models can capture the resource assignment that is required during each process

step. Therefore process models directly support the identification of responsible



persons in the context of data collection. This advantage of process modeling gets

more obvious when a complex inter-organizational process needs to be measured.

(3) It is important to identify when the data have to be collected. For example,

often product data can be collected only if the product has reached a stable state.

b. Purpose: To model the process that leads to recognizing the exact process

activities, their order of execution and the resources involved. The modeling

makes sure that the processes are set up correctly.

c. Procedure: Use data collection methods to model each process.

d. Output: Process models.

6. Derive performance metrics per process:

a. Rationale: Having the process performance objectives defined for each process,

the very next step according to Kueng (2000) is to derive the performance

metrics. Defining process performance metrics is to assess the process

performance objectives.

b. Purpose: To elicit the process performance metrics.

c. Procedure: Follow the steps explained in PEPM approach.

d. Output: The set of process performance metrics for each of modeled processes.

7. Evaluate performance metrics:

a. Rationale: Eliciting of process performance metrics is more a subjective process.

Therefore the defined process performance metrics should be verified and

validated. The verification is to determine the quality of the performance metrics.

The validation of the performance metrics is whether they assess the process

performance objectives.

b. Purpose: To verify and validate the derived performance metrics.

c. Procedure: Verify and validate performance metrics against PEPM approach

solutions.

d. Output: The process performance metrics are verified and validated.

3.4 step 1: E-procurement chain analysis

The performance measurement of the e-procurement chain is the objective of PEPM

approach. Based on section 2.1, there are different business models and forms for e-



procurement systems and chains. Hence the structure and scope of the belonging chain

differs when the business models and the enabled e-procurement businesses are of different

models and types. Therefore, the first step of PEPM approach is to analyze the e-

procurement system and its underlying e-marketplace model. E-procurement system

analysis and identification of the involved parties in the chain are the objectives of

performing this step.

Informal interviews with the chain management and reading the available documentations

of e-procurement system under study are the means for applying this step. The application

of this step should outputs factors stated in table 4.

1. E-procurement type The e-procurement forms and the underlying e-marketplace type

are identified for the e-procurement system under study.

2. Chain structure The parties and systems are clearly identified and a sketch of the

total chain is developed.

Table 4: PEPM's outputs of executing step 1

3.5 step 2: Identifying and measuring e-services

In this step of PEPM approach the offered e-services are identified and then the set of

relevant performance metrics are introduced.

The identification of e-services is carried out by analysis of the e-procurement system,

reading the available documentations and informal discussion with the experienced people.

The type of e-procurement system and the underlying e-marketplace model is the main

guidance to this identification step.

Once the e-services are identified, the performance of the services can be measured. Dumas

et al. (2001) argue that the quality of service is an auditable aspect of most services. They

express that consumer’s expectations of the service being requested and the level of

commitment that the service provider has for completing the service request are the two

main dimensions that characterize the quality of service domains. Further, Jansen-Vullers

and Reijers (2003) discuss the performance aspects for e-services. Based on these two

studies, the performance aspects for evaluating the e-services are: (1) accessibility: a measure

of the access/uptime of the service, (2) reliability: a measure of the probability of success of

the transactions involved in the service provisioning, (3) conformance: the probability that a

service provider’s service level agreement is fulfilled in a particular situation, (4) cost: a

measure for expenditures made for enabling the service, (5) timeliness: a measure of the

speed of service execution, and (6) quality: a measure for user’s satisfaction and loyalty of the

service.



This implies that the identified e-services of e-procurement system can be controlled by

these performance aspects. Further, with the help of the provided set of processes in ITIL,

the set of relevant processes can be selected. Using the syntax and semantic of the value-

focused thinking, the top level objectives (performance aspects) can be linked to the

corresponding service management processes obtained from the set of ITIL processes. In

figure 18 the quality service performance aspects are linked to the relevant ITIL processes.

For the attached ITIL processes to each performance aspects of e-services the set of

performance metrics needs to be produced. The provided set of performance metrics for

each ITIL process, available in appendix C, can be used for the initial identification and

definition of performance metrics for each quality of service performance aspect.


services


services


services


of services


services



- Continuity Management - Service Evaluation

- Process Evalutation

- Improvement Initiatives





- Process timeliness

- Resource timeliness

Figure 18: Controlling e-services of e-procurement system

Thus the e-services of an e-procurement system can be measured using the provided

structure in figure 18. Depending on the nature and type of e-service, the ITIL

corresponding performance metrics may be adapted to the situation of the e-procurement

system under analysis through discussion rounds with chain management of e-procurement

system, see figure 19 for quality of service performance metrics.

The only quality of service performance aspect that cannot be addressed using the existing

set of performance metrics of ITIL is the timeliness. There are multiple parties interacting

with the enabled business processes, and the speed of process depends on the behavior of



participants within the business process. Therefore the timeliness performance aspect can

be measured when the process is modeled where all the interactions and dependencies can

be indicated. Thus, to define performance metrics that address the timeliness issue of the e-

service, the enabled process needs to be measured. In the later steps of PEPM approach, it is

explained how a general process can be measured (adapted from Kueng (2000)).


- Continuity Management

- Service Evaluation

- Process Evalutation

- Improvement Initiatives




- Service availability- Number of service Interruptions- Duration of service interruptions- Availability Monitoring- Availability Measures

- Business processes with continuity agreements- Gaps in disaster preparation- Implementation duration- Number of disaster practices- Number of identified shortcomings during disaster practices

- Services covered by SLAs- Services covered by OLAs/Ucs- Monitored SLAs- SLAs under Review- Fulfilment of Service Levels- Number of Service Issues

- Adherence to Budgeting Process- Cost-/ Benefit Estimation- Post Implementation Review- Adherence to Approved Budget- Adherence to Project Resources- Proposals for Cost Optimization

- Number of Customer Complaints- Number of Accepted Customer Complaints- Number of Customer Satisfaction Surveys- Percentage of Returned Questionnaires- Number of Service Evaluations- Number of Identified Weaknesses

- Number of Process Assessments- Number of Process Evaluations- Number of Identified Weaknesses

- Number of Improvement Initiatives- Number of Completed Improvement Initiatives

ITIL

qu

alit

y o

f se

rvic

e p

roce

sses

Figure 19: performance metrics for QoS control

3.6 step 3: Identifying supporting processes and operations

Besides the e-services of e-procurement system, there are supporting processes that are there

to enable, support and maintain the e-services. A large part of the supporting processes is

aimed at maintaining the means of production (Reijers, 2002). ITIL identifies several

supporting processes for each of its identified domains (see section 2.4.4). The identified

ITIL domains can be used for introducing the sub goals for controlling supporting

operations. Figure 20 illustrates the link between ITIL processes and identified sub goals.







- Service Catalogue Management

- Architecture Management

- Supplier Management

- RiskManagement

- Security Management

- Capacity Management

- Change Management

- Project Management

- Release Management - Service validation & testing

- Application developement

- Configuration Management

- Knowledge Management

- Incident Management

- Problem Management

- Operations Management

- Access Management

- Facilities Management


- Service Portfolio Management

Figure 20: Controlling supporting operations of e-procurement system

To apply this step to e-procurement situation the following sub steps must be carried out: (1)

service portfolio management process needs to be adapted to e-procurement situation and

ultimately must be replaced by processes that their objective is to manage and promote

service portfolio of e-procurement system; (2) The responsible group (s) for execution of

each process must be determined. The sources of groups are identified in first step of PEPM

approach; (3) there are many such kinds of processes in large and widely stretched e-

procurement systems but the objective here should be to keep the number of key processes

to a manageable yet useful level. Therefore selection criteria are needed for these supporting

processes that are critical for the operational control of the e-procurement system.

The operational control of the e-procurement system contains the supervision of parties and

systems across the chain. Hence the criterion for the inclusion of the supporting processes

for the performance measurement program is to consider those processes that span across

multiple parties and systems. The optimal performance of these cross-party processes

depends not only on the individual performance of the participating parties and systems but

more importantly on the collaborations and interactions between the parties and systems.

The second criterion is the outsourcing issue. In case some of the processes are outsourced

the measurement of these processes are not necessary. These kinds of processes can be

controlled by defining clear service level agreements.



Thus for the application of this phase, the set of to be measured processes can be identified

by filtering them against the defined selection criteria. The set of obtained critical supporting

processes can then be measured by using the next steps of PEPM approach.

3.7 step 4: Defining process performance objectives

The procedure of defining performance metrics are based on the steps explained in the

Kueng (2000) paper.

He mentions that the first step for the process performance measurement is to define

process performance objectives. But how this step is carried out is not explained. Therefore

the PEPM approach introduces a structural method for this matter.

The performance objectives of a process can be determined through partial using of the

GQM method and combining the studied performance metric categorizations. In order to

provide a structural approach for the identification of the process objectives, the steps to

discover the ‘Goal’ aspect of GQM is used. The GQM introduces purpose, issue, object, and

viewpoint dimensions for Goal aspect (see section 2.4.1 for more details). The approach

however does not specify the possible issue categories. In section 2.3 different classifications

of performance metrics are reviewed. Among these the University of California (UC)

classification provides the extensive performance categories that are applicable for the issue

dimension of Goal of GQM method. In order to complete the set of performance metrics

types, the cost category, mentioned in the Devil’s quadrangle classifications, is added to the

UC categories. For the object dimension in GQM method, the products, processes, and

resources are mentioned as the possible types.

The viewpoint dimension is the chain management of e-procurement business in PEPM

approach.

To summarize this step, the objectives of each process are obtained by filling in the table 5

iteratively for each process:

Purpose Improve, Detect, Discover, Reduce, …

Issue Effectiveness, Quality, Timeliness, Productivity, Safety, Cost

Object Process, Resource, Product

Viewpoint Chain management

Table 5: PEPM process objective structure

Interviews can be carried out for the application of this step of the approach. Table 5 can be

used as the tool while taking interview for obtaining the process performance objectives.



3.8 step 5: Process modeling

The process will be modeled in order to visualize and overview the process steps, their order

of execution and involved resources.

Business process modeling is the activity of representing the processes so that it can be

analyzed and improved. Business process modeling is typically performed by business

analysts and managers who are seeking to improve process efficiency and quality (Ko,

2009). It outlines the involved parties and systems in the process and determines the order

of the activities. Therefore to measure the performance of a process, the modeling is

important.

There are several modeling languages. These languages standardize the way processes are

modeled. Event-process chain (EPC), business process modeling notation (BPMN), and

unified modeling language (UML) are some of the well-known process modeling languages.

In the PEPM approach the BPMN modeling language is selected. It is a graphical

representation for specifying business processes in a workflow. BPMN was developed by

Business Process Management Initiative (BPMI), and is currently maintained by the Object

Management Group since the two organizations merged a few years ago.

The main objective of BPMN is to support business process management for both technical

users and business users by providing a notation that is intuitive to business users yet able

to represent complex process semantics. In addition, BPMN models can be mapped to

constructs of execution languages. These advantages were the main reasons for using BPMN

as the modeling language in the PEPM approach.

Thus BPMN is selected as the modeling language in order to obtain a structured overview of

the process which simplifies the extraction of performance metrics for distinct process steps.

For each model, the process input, output, and description are detailed.

In appendix A, the core constructs of the BPMN language are illustrated.

The performance measurement is carried out from the chain management point of view.

Therefore the process is modeled in proportion to the process objectives put by the chain

management. This implies that the processes are modeled with different level of details in

order to avoid unnecessary complexity in the models.

3.9 step 6: Deriving performance metrics per process

The procedure of deriving performance metrics and their documentations in PEPM

approach are described in this step.



Performance metric identification

Kueng (2000) suggests deriving the performance metrics from the scratch based on the

stakeholder’s needs. However the proposed performance metrics method is applicable to any

type of organization and to any situation, the process of deriving from the scratch is very

time-consuming; especially when there are many complex processes (in case of e-

procurement performance measurement) to be measured. In addition, the proposed method

requires extensive knowledge and experience in process performance measurement because

it is based on number of meetings and does not follow any specific rule or solution.

Therefore the PEPM approach tries to use from the existing generic sets of performance

metrics identified by the industry standards and performance measurement related studies.

The available set of metrics in industry standards and process-oriented approaches is used as

a reference in order to save time for the derivation of performance metrics for each process.

In PEPM approach, primarily, ITIL performance indicators are used and are enlisted in

Appendix C.

PEPM approach provides the following process for deriving performance metrics from an

existing set and adapting to a specific situation: (1) a performance metric is selected for a

process if it can be mapped to one of process performance objectives, (2) based on GQM

definitions a performance metric targets either a product or a process or a resource.

Therefore target of performance metric should be identified and then adapted with the help

of BPMN model of the process, (3) performance metrics characteristics (name, definition,

unit, etc.) can then be documented according to the format explained in following section.

The use of given performance metric derivation process in PEPM approach does not cover

defining a performance metric from scratch. In case a new metric has to be defined, the

Kueng (2000) method can be used.

Performance metric documentation

Seel and Thomas (2007) document performance metrics in a specific table. Performance

metric definition, unit of measure, and person responsible for collecting the data are the

properties in their table. In order to address which process performance aspect is targeted, in

PEPM approach extends Seel and Thomas (2007) performance metrics with the Devil’s

quadrangle (DQ) dimension. Reijers and Mansar (2005) explain that these dimensions are

suitable for the process performance measurement. Thus the documentation of

performance metrics according to PEPM approach is the format described in table 6.



Performance metric Explanation Unit Responsible DQ-Dimension

Name of

performance

metric

Explicit

definition of

performance

metric is

provided here.

Unit of measure

is determined,

e.g.:

#: Number

%: Percentage

€: Finance

The party in the

chain responsible

for the actual

measurement

To denote which

domain is

measure belongs

to: Quality, time,

flexibility, and

cost

Table 6: PEPM performance metric documentation

3.10 step 7: Performance metrics verification and validation

The last step in PEPM approach is to verify and validate the derived performance metrics per

process.

There are two steps for verifying and validating the produced performance metrics of each

process in PEPM approach. The quality test is to verify and relevancy to validate the

performance metrics.

Performance metrics verification

University of California (UC) performance measurement approach provides a specific test to

determine the quality of a particular performance metric, the so-called S.M.A.R.T. test. This

test verifies whether the define performance metrics meet certain quality criteria. The quality

criteria of SMART test are explained in table 7.

S = Specific

clear and focused to avoid misinterpretation. Should include measure assumptions and definitions and be easily interpreted.

M = Measurable

can be quantified and compared to other data. It should allow for meaningful statistical analysis. Avoid "yes/no" measures except in limited cases, such as start-up or systems-in-place situations.

A = Attainable achievable, reasonable, and credible under conditions expected.

R = Realistic fits into the organization's constraints and is cost-effective.

T = Timely doable within the time frame given.

Table 7: UC performance metric quality criteria



The SMART test provides an explicit procedure for examining the performance metrics. The

ultimate objective of SMART test is to improve the quality of performance metrics.

The first two quality criteria, ‘specific’ and ‘measurable’, verify whether clear explanation,

target of performance metric and an appropriate measurable unit are defined for the

performance metrics. The verification of these two criteria can be done when the

performance metrics are defined. But verifying the rest of quality criteria of SMART test,

‘Attainable’, ‘Realistic’, and ‘Timely’ can be examined when the performance metrics are

being implemented. Since PEPM approach’s focus is to produce set of performance metrics

and the approach does not go into the implementation of performance metrics, these quality

criteria cannot be verified.

Concluding that in PEPM approach, the defined performance metrics for each process are

verified against the ‘specific’ and ‘measured’ quality criteria. The rest of the quality criteria

can be verified when the performance metrics are implemented.

Performance metrics validation

The derived process performance metrics should assess the defined process performance objectives. Mapping performance metrics to objectives of a process reflects to what extent the objectives are assessed. On this way, either the irrelevant performance metrics or unmeasured process performance objectives are detected. In PEPM approach, the derived process performance metrics can be mapped to the process objectives using the table 8.

Measures \ Objectives Objective 1 Objective 2 …

Performance metric 1

Performance metric 2

…

Table 8: PEPM performance metric evaluation

Although the PEPM approach tries to validate the derived performance metrics, but Folan

and Browne (2005) emphasize that the procedure of selecting a particular performance

measure is, to a certain extent, a subjective process. It often involves top management sitting

around a table and choosing one measure from a number of alternatives. They indicate that

the process of deriving performance metrics is an iterative process that should happen from

time to time, depending how the business and its scopes changes.



3.11 Conclusions

A process-oriented performance measurement approach (i.e. PEPM approach) is developed

to assess the performance of e-procurement chains. The PEPM approach consists of several

steps. Each of these steps are developed by partially using the solutions of the existing

approaches and methods. GQM method and Kueng-approach, industry standards such as

ITIL, BPMN modeling language are examples of the used approaches in developing each

step of PEPM approach. Figure 21 illustrates an overview of PEPM approach and its steps.


its chain



Step 4: Set process




performance metrics


metrics


Figure 21: PEPM approach



4. PEPM application on Raboshop

The constructed approach is evaluated by applying it to Raboshop as an e-procurement

system. Through this phase the applicability and practicality aspects of the developed

method is justified. The required and retrieved data and information from Raboshop

situation and its chain are gathered by the use of collection methods mentioned in section

1.3.5.

4.1. Step 1 of PEPM approach

The Raboshop was analyzed and the structure of the Raboshop chain was identified.

4.1.1 Raboshop and E-procurement

Raboshop is an e-procurement system that incorporates e-ordering and e-contract types.

Raboshop supports the contract-to-pay process, i.e. from contracting until and including

billing. Thus Raboshop is a type of e-procurement system that automates the operational

part of procurement process defined by van Weele (2003), section 2.1.2. The relationship

between contract management, purchase orders, invoices and payments results in a so-called

closed loop between the tactical and operational procurement process, see figure 22.

Approve

Requisition

Invo

icin

g

Monitor

Receive

Order

Assi

gn

Stra

tegi

ze

QualifyAw

ard

Negotiate

Contract

Analyze

Procurement

Sourcing

Figure 22: Closed loop: relation between contract, orders, and invoicing (RCI, 2009)

In addition, Raboshop offers services for the suppliers of the products as well such as

catalogue management. Therefore, the fact that Raboshop provides services for both buyers

and suppliers implies the neutral type of Raboshop as an e-marketplace.



4.1.2 Raboshop chain structure

Raboshop consists of several underlying systems and multiple parties are interacting with

this system (Raboshop chain). These are explained in this step.

Raboshop as an e-procurement system is composed of several underlying IT-systems. In

table 9 a high level definition of the underlying system is presented.

SAP

Raboshop is a SAP-oriented information system composed of several SAP

modules. SAP SRM, SAP MDM are the most important modules.

FA

This is the system used for electronic invoicing of the orders. This system

interacts with Raboshop SAP modules.

Table 9: Raboshop underlying IT systems

The Raboshop systems are supported by several IT-provider groups. Table 10 explains core

parties in Raboshop chain.

Party Complete name Role

RCI Rabobank

Concern Inkoop

RCI is the front side of Raboshop chain. It is

responsible for service portfolio management and

business development. This group monitors all the

other groups within the chain as well.

CRG Control Rabobank

Group

Functional management of Raboshop is the

responsibility of this group.

GICT Group ICT Application management and development

management are the responsibilities of this group.

Table 10: Raboshop core parties

In addition to the core parties of Raboshop, there are other groups that interacting with the

system. Table 11 enlists these parties.

Party Role

Users

A large group of authorized users can procure their products using

the Raboshop e-procurement system. Local banks and affiliated

institutions are examples of authorized users of Raboshop.

Product owners This group is responsible for offering products and services in



(PO) Raboshop catalogue.

Suppliers (S) This group is responsible for supplying the actual Raboshop products.

Logistics Service

Provider (LSP)

This party is an intermediary transporter for Raboshop stock

products.

Cross-dock Transporting of some of products is carried out via this party.

Table 11: Raboshop interacting parties

The total overview of the participants and their relationships within the Raboshop business

are modeled in figure 23.

CRG

GICT

Users

Raboshop System

Product Owners

Suppliers

LSP

Cross-dock

Transport

Sellers

Buyers

RCI

Raboshop Business

Intermediaries

Figure 23: Involved parties and their relationships in Raboshop chain



The Raboshop chain management (i.e. RCI) supervises, communicates, coordinates, directs,

and facilitates parties and underlying systems in order to ensure the continuous business

operability and to plan the future improvements and business expansions.

4.2 Step 2 of PEPM approach

The objective of this step is to discover the Raboshop major e-services and measure them.

Raboshop e-services:

As identified in step 1, Raboshop is a neutral e-marketplace providing services for both

buyers and sellers.

Bearing this in mind, the following services are provided by Raboshop (table 12):

Process name Description Target groups

Catalogue buying

process

This service enables the users of Raboshop to

procure products and services available in

Raboshop catalogue.

Buyers, Sellers

Free order process

This service enables the users of Raboshop order

the products and services not available in Raboshop

catalogue.

Buyers, Sellers

Contract labor This service partially facilitates the process of

external hiring process. Buyers

Operational contract

management

Through this service, contract information can be

recorded in Raboshop for online use. Sellers

Catalogue

Management

Product catalogues are managed through this

service. Sellers, Buyers

Replenishment

Management

This service ensures on time replenishments for on

time delivery of Raboshop stock products. Sellers

Product category

management

This service takes care of changes to the product

categories (Product category is a code with which

products and services with similar characteristics

can be summarized in a specific group of goods).

Buyers, Sellers

Reporting This service provides reports of procurement data. Buyers, Sellers

Table 12: Raboshop e-services



Measurement of Raboshop services:

The identified Raboshop services are measured based on the identified quality of service

performance aspects explained in step 2 of PEPM approach. ITIL provides set of

performance metrics for the measurement of quality of Raboshop e-services. The

explanation of the corresponding processes and performance metrics are attached in

appendices C and D. Performance metrics that address the timeliness performance aspect of

Raboshop services are identified in the process measuring stage of PEPM approach (i.e.

steps 4 – 7).

4.3 Step 3 of PEPM approach

The objective of this step is to discover the Raboshop supporting processes.

Raboshop supporting processes:

The sub steps one and two were applied on figure 20 for Raboshop situation and figure 24 is

obtained.

Raboshop Controlling supporting operations




- Change Management (RCI/CRG/GICT)

- Project Management (RCI)

- Release Management (CRG/RCI) - Service validation & testing (CRG/RCI)

- Application developement (GICT)

- Configuration Management (GICT/CRG)

- Knowledge Management (CRG)


- Business Developement (RCI)

- Buyers Relationship Management (RCI)

- Sellers Relationship Management (RCI)

- Training Management (CRG/RCI)

- Service Catalogue Management (RCI)

- Architecture Management (CRG/GICT)

- Supplier Management (CRG/GICT)

- RiskManagement (CRG/GICT)

- Security Management (CRG/GICT)

- Capacity Management (CRG/GICT)

- Incident Management (RCI/CRG/GICT)

- Problem Management (RCI)

- Operations Management (CRG/RCI)

- Access Management (CRG/RCI)

- Facilities Management (CRG/GICT)

Figure 24: To be controlled Raboshop supporting processes



The sub step there was then applied. Cross-party and outsourcing criteria were applied to list

of supporting processes. List of critical supporting processes are obtained and explained in

table 13.

Process Name Description

Incident management

This process aims to restore normal service operation as quickly as

possible and minimize the adverse effect on business operations,

thus ensuring that the best possible levels of service-quality and –

availability are maintained.

Problem management

The process aims to resolve the root causes of incidents and thus

to minimize the adverse impact of incidents and problems on

business.

Change management This process explains the standard procedures and methods that

are used for well-structured and efficient handling of all changes.

Release management

This process regulates the implementation of any approved change

and focuses on the protection of the live environment and its

services through the use of formal procedures and checks.

Project management

This process is about a temporary endeavor, having a defined

beginning and end, undertaken to meet unique goals and

objectives, usually to bring about beneficial change or added value.

Training management

Raboshop customers and all the employees working with

Raboshop need specific training. This process takes care of

appropriate training material for different target groups.

Table 13: Raboshop critical supporting processes

The measurement of these supporting processes is done in the following sections which

explain the process performance measurement of the processes with the help of PEPM

approach.

4.4 Steps 4 – 7: Process objectives, models, metrics, and validations

Steps 4 – 7 of PEPM approach are bundled in this section. This is done in order to bundle

the results of performance measuring for each process and create a better overview of the

steps taken.

Due to combining multiple steps of the PEPM approach, the following format (figure 25) is

used for each process.



Figure 25: second phase of PEPM approach

Based on the results of the previous steps of PEPM approach on Raboshop situation, the

processes are measured based on their type. If the process is a Raboshop service the

timeliness factor, as the only performance aspect, is measured otherwise all the performance

aspects are considered. Further, the following abbreviations are used while modeling the

processes (table 14).

SC Shopping Cart RFC Request For Change

PO Purchase Order UAT User Acceptance Test

POR Purchase Order Response CRG – CA CRG Cost Accounting

POC Purchase Order Confirmation CRG – BI CRG Functional manager

GR Goods Receipt FAB Functioneel Applicatie Beheer

ERS Electronic Receipt Settlement TAB Technisch Applicatie Beheer

CRG – FA CRG Financial Accounting OLA Operational Level Agreement

UC Under-pinning Contracts SLA Service Level Agreement

Table 14: List of abbreviations

4.4.1 Catalogue buying process

Catalogue buying process is the main service of the Raboshop e-procurement system.

Multiple parties and multiple systems are executing different steps of this process.



Process performance objectives:

1. To improve cycle time of catalogue buying process.

2. To improve response time of involved resources.

Process input: The authorized clients of Raboshop system can login the Raboshop web-

oriented portal. They navigate the available product catalogues and they create a shopping

cart by selecting their desired products or services. The catalogue buying process is initiated

by submitting the shopping cart.

Process output: The product is delivered and the corresponding invoice is issued or the

shopping cart is rejected.

Process description: The submitted SC is automatically controlled and based on the required

criteria it needs to be completed, approved or a purchase order is created directly. Depending

on the purchase value limit of the buyer or type of the product, the manager of buyer or

product owner are prompted to further process SC. If all steps are okay, a PO is created and

sent to the supplier. Supplier responses and creates a POC and delivers the product either

directly or via a cross-dock. After the GR confirmation, the purchase order is invoiced and

the process workflow of the catalogue buying process is terminated.

Process model:

En

duse

r M

anag

erR

abos

hop

En

duse

rS

upp

lier

FA

Sent for approval

Approval needed

Rejected

Confirm goods receips (GR)

Process PO (POR)

Cro

ss

dock

er

PO confirmation (POC) creation

ok

Nok

Send goods

To Cross docker

Invoicing

+

Execute invoicing process (ERS)

Completion needed

Submit shopping cart

(SC)

Approved

Rejected

To Enduser

+

Start return delivery process

Invoicing

Receive decision

Receive goods

Purchase order (PO) creation

Receive goods/message

Rejected

Send goods

Sent for Completion

Confirm completed SC

Pro

duct

Ow

ner

+

SC Completion process

+

SC Approval process



Performance measure Explanation Unit Responsible DQ-

dimension

SC-PO response time

The average time elapsed from submitting

the shopping cart till creation of the

purchase order.

Days CRG Time

SC-SC response time


the shopping cart till approving the

shopping cart (for special types of

products).

Days CRG Time

PO-POR response time The average time elapsed from receiving

PO till approving it by the supplier. Days CRG Time

POR-POC response time The average time elapsed from POR till

delivering goods. Days CRG Time

POC Cross-dock average

time

The average time elapsed from creation of

POC and delivery from cross-dock. Days CRG Time

POC – GR time

The average time between POC creation

and GR in case of deliveries more than the

specified amount (responsiveness of the

receiver)

Days CRG Time

GR – ERS time

The average time between GR and ERS.

This can be used to check if all approved

GR’s are on time invoiced and to discover

wrong settings.

Days CRG Time

GR – Return goods time

The average time between GR creation and

Approve goods return in case of the return

goods process. It can be used to check if all

return requests are handled on time.

Days CRG Time

SC – ERS time The average time elapsed from submitting

the shopping cart till invoicing. Days CRG Time

Time between SC – ERS

with SC approval



Time between SC – ERS

without SC approval





Performance metrics verification and validation:

- All the performance metrics pass the quality factors of PEPM approach. The definitions of performance metrics are specific and measurable.

- The performance metrics are mapped to the process objectives to show their

relatedness.

Measures \ Objectives 1. Process

Timeliness 2. Resource

Response time SC-PO response time

x

SC-SC response time x

PO-POR response time x

POR-POC response time

POC Cross-docker average time x

POC – GR time x x

GR – ERS time x

GR – Return goods x

Time between SC - ERS x

Time between SC – ERS with SC

approval x x

Time between SC – ERS without

SC approval x

4.4.2 Free order process

Raboshop provides the possibility for its clients to order products which are not present in

Raboshop catalogue. This service is called free ordering process.


1. To improve the timeliness of the free order process

2. To improve the response time of the resources

Process input: The process is initiated as soon as a non-catalogue product is ordered.



Process output: The order is either accepted and product is delivered or is at some stage

rejected and the buyer is notified.

Process description: SC for requested free order is sent to the operational purchaser (product

owner) for the eventual order completion (e.g. assigning a price). Thereafter the user

confirms completed order and his manager approves order and PO is created. PO is then

printed out by requestor and signed by his manager. The signed PO is then sent to the

relevant supplier. The supplier confirms the PO and delivers the product. In the last step the

invoicing is processed.

Process model:

En

du

ser

Man

ager

Rab

osh

op

En

du

ser

Su

pp

lier

FA

Yes

Confirm goods receips (GR)

ok

Nok

Invoicing

+

Execute invoicing process (ERS)

Submit Freeorder

SC

+

Start return delivery process

Invoicing

Sent for Completion

Pro

du

ct

Ow

ner

+


No

Print out PO and signature

Send PO to supplier

Deliver order

Complete? Recieve goods

Sent for Approval

+


+

Signature process

Performance

measure Explanation Unit Responsible DQ-Dimension

Cycle time free

order

Average total time spent from

placing a free order till and

including invoicing.

Days CRG Time

SC – PO

completion

response time

Average time between receiving SC

and completing the SC by the

product owner.

Days CRG Time

SC – PO approval Average time between receiving the Days CRG Time



Performance metrics verification and validation: - All the performance metrics pass the quality factors of PEPM approach. The definitions

of performance metrics are specific and measurable.

- The performance metrics are mapped to the process objectives to show their relatedness.

Measures \ Objectives 1. Process Timeliness

2. Resource Response time

Cycle time free order x

SC – PO completion response

time x

SC – PO approval time x

PO – Delivery time x x

Invoicing time x

4.4.3 Operational contract management

Alongside ordering possibilities, Raboshop offers services for e-contracting form for procurement process. Process performance objectives: (1) to improve cycle time of contract registration process. Process input: A request for contract registering or contract change is submitted. Process output: Either the request is accepted and processed or the request is rejected. Process description: The conditions for the request of creating a new contract or changing the exiting one are checked and a decision will be made. All the steps of contract registering and contract updates are monitored. Process model:

time agreed SC and PO approval.

PO – Delivery

time

Average time from receiving the

PO by supplier and delivering the

goods.

Days CRG Time

Invoicing time Average time for invoicing process. Days CRG Time



Con

trac

t Man

ager

sR

CI

+

Prepare contract

registration

+

Finalize contract

registration

+

Contract control

and release

+

Commuicate

contract

Performance metrics verification and validation: - All the performance metrics pass the quality factors of PEPM approach. The definitions

of performance metrics are specific and measurable.



Time to record a contract x

4.4.4 Replenishment management

Raboshop has stock and non-stock products. Stock products are kept by LSP in order to

avoid backordering and delivery delays for customers of Raboshop. Product owners monitor

level of stock for stock products and if necessary they create replenishment orders.

Process objective:

1. To improve cycle time of replenishment process.

2. To improve resource response time.

Process input: The replenishment order is submitted.

Process output: The goods are delivered to logistics service provider.

Performance measure Explanation Unit Responsible DQ-Dimension

Time to record a

contract

Average time to record a contract in

Raboshop. Days RCI Time



Process description: The received replenishment order is processed in FA4U system manually.

Thereafter it is sent to Raboshop SRM for sourcing and purchase order creation. Logistics

service provider prepares itself for the new delivery and supplier delivers the ordered

products. Stock levels of the products are created and sent to product owners for the task of

stock level monitoring.

Process model:

RC

IP

rodu

ct O

wn

erSu

pplie

rLS

P

Create stock overview

Send stock overview

+

Monitor product stock

level+

Create & send replenishment

request

Control & process request

+

Goods delivery process

+

Goods receipt process


Time for

replenishment

process

Average time for of completion

replenishment process. Days RCI Time

Time to control and

process order

Average time spent to control

and process the replenishment

process.

Days RCI Time

Time to deliverying

goods

Average time from receiving

replenishment order till

delivering goods to LSP.

Days Supplier Time

Time to create stock

overview

Average time to create stock

overview. Days RCI Time



Performance metrics verification and validation: - All the performance metrics pass the quality factors of PEPM approach. The

definitions of performance metrics are specific and measurable. - The performance metrics are mapped to the process objectives to show their

relatedness.


2. Resource response time

Time for replenishment

process x

Time to control and process

order x

Time to send goods x

Time to create stock overview x

4.4.5 Catalogue management

This process aims at efficient and effective establishment, management, and utilization of

product catalogues present in Raboshop.

Process objectives:

1. To improve cycle time of process.

2. To improve response time of resources.

Process input: Product owner submits a request regarding a new/changed/removed product

catalogue.

Process output: The request is either rejected or is applied in the respective Raboshop

product catalogue.

Process description: After request submission, it is controlled. Upon that it is checked

whether the attached-file is complete. Then the file is loaded into the system and product

owner is asked for approval. The changes are implemented permanently after product owner

approval.

Process model:



Send product data request

Pro

du

ct o

wn

erR

CI

Publish in Raboshop SRM

Check the file

Insert product data in FA4U

stock

Enrich product data in SAP

MDM

Non stock

Further enrich product data

Attach to a contract


definitions of performance metrics are specific and measurable. - The performance metrics are mapped to the process objectives to show their

relatedness.

Performance measure Explanation Unit Responsible DQ-quality

Time to process the

stock product requests

Average cycle time from receiving

the request for stock product till its

complete processing.

Days RCI time

Time to process the

non-stock product

requests

Average cycle time from receiving

the request for non-stock product

till its complete processing.

Days RCI time

Change/Creation

products –

Approving/Releasing

products time

The average time between

creation/change products and

approving/releasing the products.

It is to measure the progress of

handling new products.

Days CRG Time

Time to process

catalogue request with

product owner

Average time when product owner

is involved in completion of

catalogue request process

Days RCI Time

Time to process

catalogue request

without product owner

Average time when product owner

is not involved in completion of

catalogue request process

Days RCI Time




2. Resource response time

Time to process the stock product

requests x

Time to process the non-stock product

requests x

Change/Creation products –

Approving/Releasing products time x

Time to process catalogue request with

product owner x

Time to process catalogue request

without product owner x

4.4.6 Product category management

Raboshop products are divided into several product categories and each of these product

categories (PC) are bound to specific contracts with the corresponding suppliers.

Process objectives:

1. To improve cycle time of process.

2. To improve response time of resource.

Input: A request for changing an existing PC or for adding a new PC initiates this process.

Output: The request is either rejected or is applied in the Raboshop.

Description: After process is initiated, a formal request is created and sent for further

completion. Thereafter the request is checked and if everything is okay the complete form is

sent for implementation. A system format form is created and sent back for the completeness

and correctness. At last the product category is created or the existing one is changed.

Process model:



Control request

content

RC

IC

RG

Add necessary

info

Construct a

formal RFC for

PC

GIC

T

Construct an

internal request

Send RFC to

CRG-BI

Control RFC and

complete

Pro

du

ct O

wn

er

Send a request for

change/new PC

Create/

Change PC

Control new/

changed PC

Receive feedback

request

Not ok

Ok


Time to change an

existing category

Average time from receiving a PC

change request till completing the

process.

Days RCI Time

Time to create a new PC Average time to create a new PC. Days RCI Time

Time to construct a

formal RFC

Average time to construct the required

RFC in this process. Days RCI Time

Time to control RFC

and complete

Average time to control and complete

RFC. Days CRG Time

Time to create/change

PC in the system

Average time to implement changes or

add new pc in system Days GICT Time

Time to construct a PC

request

Average time to construct a PC request Days RCI Time

Time to control PC

request

Average time to control a PC request Days CRG Time




definitions of performance metrics are specific and measurable.


relatedness.

Measures \ Objectives 1. Process

Timeliness 2. Resource

response time

Time to change an existing category x

Time to create a new PC x

Time to construct a formal RFC x

Time to control RFC and complete x

Time to create/change PC in the system x

Time to construct a PC request x

Time to control PC request x

4.4.7 Incident process

In case any kind of problem or interruption of services is face, customer service can be

reached.


1. To shorten cycle time of incident process.

2. To improve response time of resources.

3. To improve the effective response for an incident.

4. To improve the efficiency of incident resolution by distributing the incidents

through the involved parties correctly.

Process input: Submitting an incident triggers incident process.

Process output: Incident is either resolved or unresolved.

Process description: After the incident is submitted, customer service categorizes the

problem and tries to solve the incident right away. If it cannot be solved, incident will be

sent to CRG. Either the problem will be solved there or it will be sent to GICT for further

handling. Some of the incidents related to functional question can be answered by RCI.



Process model:

0th Line Support

Special Line Support

1st Line Support

2nd Line Supoort

+

Resolve Incident

RC

IIn

cid

ent

Man

agem

ent

Pro

cess

RC

I-C

ust

om

er

Ser

vice

CR

GG

ICT

+

Resolve Incident

+

Resolve Incident

+

Resolve Incident

Possible triggers;- E-mail- Phone call- System malfunctioning




Incident resolution time Average time spent to resolve the

incident Days RCI Time

Number of periodic

incidents

The total number of incidents

registered by the customer service

for specific periods.

# RCI Quality

Number of incidents

buyers


submitted by buyers (users) of

Raboshop services for specific

periods.

# RCI Quality

Number of incidents

from sellers


submitted by sellers of products

and services of Raboshop chain for

specific periods.

# RCI Quality

Percentage of incidents

resolved by RCI

Proportion of number of incidents

resolved by customer service to the

total number of incidents.

% RCI Quality


resolved by CRG


resolved by CRG to the total

number of incidents.

% CRG Quality


resolved by GICT


resolved by GICT to the total

number of incidents.

% GICT Quality

Number of current

incidents

Average number of to be resolved

incidents # RCI Quality

Percentage of timely

resolved incidents

Proportion of resolved incidents

within the agreed time to the total

number of resolved incidents.

% RCI Quality

Incident occurrence rate

Plotting the number of incidents

from multiple periods to see

whether a decreasing (or

increasing) trend can be

discovered.

N/A RCI Quality



Performance metrics validation:



relatedness.


2. Resource Response time

3. Efficient response

4. Proper incident distr.

Incident resolution time x

Number of periodic incidents x

Percentage of incidents resolved

by RCI x


by CRG x


by GICT x

Number of current incidents x

Percentage of timely resolved

incidents x

Incident occurrence rate x

4.4.8 Problem management

This process looks at the roots and causes of the incidents. Through active monitoring of the

incident management process problem management is realized.

Process objectives:

1. To reduce resolution time of problem process.

2. To increase quality of the system.

3. To reduce the cost of resolving underlying incidents.

Process input: A problem is detected.

Process output: A solution for the problem is realized.

Process description: Through monitoring the incident process and internal questions of chain

partners, a problem is diagnosed. This problem will be further analyzed and discussed in

change board and upon that either a RFC can be created or the problem is isolated.



Process model: R

CI

+

Problem identification

CR

G

+

Problem diagnosis & resolution

+

Problem closure and evaluation

Performance metrics validation: - All the performance metrics pass the quality factors of PEPM approach. The definitions of

performance metrics are specific and measurable.


Performance

measure Explanation Unit Responsible DQ-Dimension

Number of

Problems

Number of problems registered by

problem management (per

category).

# RCI quality

Problem

resolution time

Average time for resolving a

problem.

Days RCI time

Number of

incidents per

problem

Average number of incidents linked

to the same problem before problem

identification.

# RCI quality

Number of

incidents per

known problem

Average number of incidents linked

to the same problem after problem

identification.

# RCI quality

Time until

problem

identification

Average time between first

occurrence of an incidents and

identification of the underlying root

cause.

Days RCI time

Problem

resolution effort

Average work effort for resolving

problems.

Days RCI cost




2. Quality of system

3. Reduce cost of incidents

Number of Problems x

Problem resolution time x

Number of incidents per problem x x

Number of incidents per known

problem x

Time until problem identification x

Problem resolution effort x

4.4.9 Change management

In order to meet the demands of customers and stay competitive in the business, the business

needs to improve and bring the necessary changes in the system. End-users, suppliers,

operational managers and other people involved in Raboshop business.

Process objectives:

1. To reduce the time of the change clearance.

2. To improve the efficiency of the important activities within the process.

3. To reduce the timeliness of activities.

4. To detect the complexity of the change process.

Process input: End-users, suppliers, operational managers and other people and parties

connected to the Raboshop business may submit their requests and desires for new changes.

These requests and desires are the primary initiative for the change process.

Process output: The submitted desire is either rejected or implemented and added to system’s

functions.

Process description: The first step in the process is to analyze and validate the submitted

request. This happens in the bi-periodic change board meetings that currently consist of RCI

and CRG group. Thereafter either the request is rejected or passed to the next phase. Upon

approval a feasibility study is initiated and the outcome will be discussed again in the change

board meeting. Again either it will be rejected or passed. Upon approval a priority will be

assigned to the request. Thereafter a formal so-called RFC will be created. This RFC is

submitted for an impact analysis. When the impact analysis is done, it will be implemented

and integrated into the system’s current features.

Process model:



Present business request

Discuss it in changebord

Prioritize Perform

feasibility study

Construct RFC

Perform impact analysis

Document the request

Rejected

Re-prioritize

RC

IC

RG

GIC

T

Make a decision

Rejected

+

Implement

Possible sources: End-users, Suppliers


Number of change

requests

The average number of periodic

submitted change requests.

# RCI Quality

Number of feasibility

studies

Number of the current feasibility

studies per a specific period.

# RCI Cost

Number of finished

feasibility studies

Number of finished feasibility

studies per a specific period of the

year.

# RCI Quality

Time for feasibility study Average time for executing a

feasibility study.

Days RCI Time

Number of RFC’s Number of RFC’s submitted for

implementation per month.

# RCI Cost

Time for impact analysis Average time spent for impact

analysis

Days CRG Time

Amount impact analysis Total amount paid for the

execution of impact analysis

studies per a specific period.

€ RCI Cost

Change Acceptance Rate Proportion of number of accepted

change requests to number of

rejected change requests per a

specific period.

% RCI Flexibility



Performance metrics validation:




2. Efficiency of activities

3. Resource timeliness

4. Process complexity

Number of feasibility studies x

Number of finished feasibility

studies x

Time for feasibility study x

Number of RFC’s x

Time for impact analysis x

Amount impact analysis x

Change Acceptance Rate x

Time for change request

clearance x

Time for RFC clearance x

Time for change request

clearance

Average time from registering a

change request till RFC creation.

Days RCI Time, quality

Time for RFC clearance Average time from RFC

submission until RFC

implementation (including impact

analysis).

Days RCI Time, quality



4.4.10 Release management

New implemented changes and additional features will be bundled together in release

version. The Raboshop releases happen multiple times per year. The process of release

needs to be carefully managed for a successful and error-free release. Any flaw in the new

version of the system may have substantial consequences in interrupting other processes.

Process objectives:

1. To improve the quality of the releases.

2. To improve the timeliness of testing and fixing errors.

3. To improve the productivity of the release process.

4. To improve test plans to reduce production issues.

Process input: A new version of Raboshop. This version includes additional features and

functions or some changes have with respect to the previous version.

Process output: A functioning version of Raboshop replaces the old one.

Process description: The first step of the release is to test the list of the implemented

additional features and the changes made in some of current system’s functionalities.

Thereafter the end-users of Raboshop will be informed about the new version. At the

same time the relevant user acceptance tests will be created (if needed) and then carried

out in order to detect potential errors and flaws in the new version. If no errors are

detected the release will go live according to the plan.

Process model:

Plan release

RC

IC

RG

Develope distributiion &

backout proceduresTest release

UAT

GIC

T

Fix errors

Fix errors

Inform the business

of new release

Carry out

release



Performance metrics verification and validation: All the performance metrics pass the quality factors of PEPM approach. The definitions of performance metrics are specific and measurable. The performance metrics are mapped to the process objectives to show their relatedness.

Measures \ Objectives 1. Process timeliness

2. Resource timeliness

3. Process productivity

Number of releases x

Number of implemented RFC’s x

Time for unit tests x

Number of identified errors x

Time for UAT x

Percentage of failed release UAT’s x

Time for error fixing x

Incidents caused by new release x


Number of releases Number of releases per year. # RCI quality

Number of

implemented RFC’s

Number of RFC’s submitted from

business side and implemented and

integrated in the new release.

# RCI quality

Time for unit tests Average total time for application

level unit testing for the new release Days CRG time

Number of identified

errors

Number of identified errors during

unit testing phase of the release # CRG quality

Time for UAT Average time for carrying out the

UA tests. Days RCI time

Percentage of failed

release UAT’s

Percentage of release components

which fail to pass acceptance tests. % RCI quality

Time for error fixing Time until re-submission of fixed

release components. Days CRG time

Incidents caused by

new release

Number of incidents attributable to

new release. # RCI quality



5. Analysis of PEPM approach

This chapter explains the analysis of the PEPM approach in both composition and

application phases. The experiences and reflections identified and perceived of the approach

are discussed here. The approach is analyzed based on its specific properties, i.e. objective

and process discovery, process performance metrics derivation, and process-oriented

approach.

5.1 PEPM versus the studied performance measurement approaches

This section analyzes PEPM approach by comparing it to the studied performance

measurement approaches. The advantages and disadvantages of the approach are discussed

5.1.1 PEPM approach versus ITIL

ITIL provides a set of processes in the domain of IT service management and associates a

number of performance metrics for each of defined processes. It focuses on processes and

lacks a clear performance measurement strategy. The PEPM approach is similar to ITIL in

the way that both are process-oriented. However, the difference is that PEPM approach has

performance measurement as his main mission. More specifically, PEPM approach uses

process-oriented approach in order to gain insights into performance aspects of e-

procurement system but ITIL focuses on managing the system and operations through

processes.

PEPM approach develops a new classification of ITIL processes by recognizing two sets of

processes: one set concentrates on performance measurement of e-services and the other set

focuses on the measurement of the supporting processes. This categorization provides a

more suitable structure for performance measurement of e-procurement chains.

The PEPM approach does not achieve its objective (i.e. operational control of e-procurement

chain through performance measurement) by only using ITIL sets of processes and metrics.

For example the timeliness performance aspect of the services cannot be addressed without

delving into the service specifics itself because the type of e-procurement services and the

involved resources determine the timeliness aspects of the services. Hence PEPM approach

contains additional procedures compared to ITIL.

5.1.2 PEPM approach vs. other studied performance measurement approaches

PEPM versus Kueng’s approach

The second studied process-oriented performance measurement approach was Kueng’s

(2000) approach. This approach explained what the necessary steps are to measure a



process. The difference between PEPM approach and Kueng’s approach is in their

objectives. The main objective of the Kueng’s approach is to provide a methodology for

structural process performance metrics identification. On the other hand, PEPM’s objective

is to provide a process-oriented methodology for performance measurement of e-

procurement chains. Therefore the difference between these two approaches is obvious,

namely, PEPM approach focus is far greater than the Kueng’s approach, operational control

of e-procurement chain where many processes are to be measured, in PEPM approach,

versus how a single process is measured, in Kueng’s approach.

PEPM versus GQM method

The GQM method is a goal question metric approach that starts by identification of goals,

development of relevant questions about the goals and ultimately definition of performance

metrics that assess the goals of the measurement. The problem with GQM method is that it

does not necessarily address how the goals are defined in a structural manner and how the

questions have to be developed. The brief explanation of how goals and questions can be

developed in Basili et al. (1994) is somewhat intuitive and demands a very deep knowledge

of the method and performance measurement experience. On the other hand, PEPM

approach is a structural process-oriented approach. The implementer of PEPM approach is

guided through the process of performance measurement and is provided with the

necessary tools and methods to apply each step of the approach. The only similarity between

PEPM approach and GQM method is when in PEPM approach the process goals are to be

defined. GQM methodology was partially used to complete this step of PEPM approach.

PEPM versus BSC method

BSC framework is a comprehensive approach that defines a balanced set of performance

perspectives. The objective of the framework is to define set of performance objective in each

of these perspectives. The defined set of performance objectives and the belonging

performance measures assess monitoring and controlling the business objectives of the

organization. The problem with this approach is that it does not necessarily address how its

steps should be carried out. Kaplan and Norton (1993) introduced a methodology for

implementing the BSC framework where a series of discussion rounds and meetings are

held with the management. Although it provides a structural way to implement the BSC

framework but it is highly dependent on the business opinions. In case the management

does not know what is needed to measure the performance, then the implementation of BSC

framework may result in unnecessary or irrelevant performance metrics. When designing

the PEPM approach, it is more emphasized on how each step should be applied in order to

provide a unified way for implementing the approach. The implementer of the PEPM

approach applies the approach by following the provided instructions. The disadvantage of

PEPM approach with respect to BSC framework is that the strategic business objectives

cannot be mapped directly to the identified process performance metrics.



PEPM versus financial performance measurement approaches

Since the objective of PEPM approach is the operational control of e-procurement chain,

little attention is paid on the financial performance of the total e-procurement business. On

the other hand, DuPont and other financial performance measurement approaches are

providing extensive methods to assess the financial performance of the organizations.

5.2 Analysis of steps of PEPM approach

Each phase of PEPM approach is analyzed in this section to discuss the strong and weak

points of the approach.

5.2.1 PEPM e-services discovery and measurement

According to the PEPM approach, after the initial step of understanding and identifying the

type of the e-procurement system and specifying the belonging structure of the chain, the e-

services of the e-procurement system are identified and measured. Performance

measurement of e-services results in gaining insights in performance level and quality of

offered e-services. Through the application of this step on Raboshop situation, there were

number of e-services detected. The Raboshop e-services had the type of either automated

(e.g. catalogue buying process) or semi automated business processes (e.g. catalogue

management). Measuring these types of e-services with the help of PEPM approach was

inline with the process-oriented aspect of the approach.

The set of identified ITIL processes and metrics for quality of service performance aspects

caused fast implementation of this step through the application phase. However

identification of timeliness performance aspect of each e-service required application of

process measurement phase of PEPM approach.

5.2.2 PEPM supporting processes identification

In addition to measuring performance of offered e-services by e-procurement system, PEPM

approach covered performance measurement of supporting operations and processes. In

order to detect which of supporting processes should be measured a couple of criteria were

introduced in PEPM approach.

The development of criteria for the selection of supporting processes was very critical for the

efficiency aspect of the approach. The ultimate objective was to keep the number of to be

measured processes to a manageable level and at the same time to achieve the goals of the

approach.

Through the application of this phase on Raboshop situation, the set of ongoing supporting

processes were detected and in order to optimize the set of to be measure processes, the



selection criteria (i.e. cross-party and outsourcing aspects) were applied. Through the

application phase, the responsibilities of supporting processes were assigned to the parties

of the chain. Hence management received an overview of the role of each party within the

set of supporting processes. Application of selection criteria identified the set of to be

measured processes. Application of this step of PEPM approach generated the critical set of

to be controlled and monitored supporting processes for Raboshop chain.

5.2.3 PEPM process performance measurement

In this stage of PEPM approach, steps 4 – 7, the discovered critical processes were

measured. To do so, the first step was to obtain objectives per process. Designing a

structured table to obtain process performance objectives ensured a unified approach for the

identification of process performance objectives. At the same time, the developed table was

the appropriate guidance on the focal points that must be taken into account when defining

the process performance objectives.

Through the application of this phase on Raboshop it was observed that clear structure of

discovering process performance objective provided a clear strategy for identifying

performance objectives. Moreover, the structure gave a unified method of documenting the

process performance objectives.

The modeling step of the process was composed of process input, output, description, and

the model in BPMN. Composing BPMN models for Raboshop situation clarified the process

participants, their responsibilities within the process and their relationship to each other.

The subsequent step in performance metric discovery phase of PEPM approach was to

derive the process performance metrics. The process performance objectives and process

model were the ingredients for the derivation of process performance metrics. In the PEPM

approach, it was recommended to use the existing set of performance metrics from ITIL and

other relevant studies and apply it to the e-procurement situation.

Through application steps for performance metrics derivation, the set of ITIL performance

indicators used for Raboshop situation were very practical, for both e-services performance

metrics and supporting processes.

The verification and validation of performance metrics in PEPM approach was based on the

quality and their relevancy. Both steps were applied for Raboshop derived process

performance metrics. Through the use of a mapping table designed in PEPM approach, the

relevancy of the performance metrics to the corresponding process performance objectives

was elaborated. The mapping table was perceived as a methodic for the validation of the

performance metrics.



As stated in section 3.10, step seven of PEPM approach, the quality testing was limited to

specificness and measurability factors of SMART test due to the focus of the PEPM

approach on performance metrics discovery and definition. The rest of SMART quality

factors are testable when the performance metrics are implemented and the relevant data are

retrieved.



6. Conclusions and Recommendations

The results of this study are summarized in this chapter. The main findings and the answer

to the main question are stated here. Thereafter the eventual recommendations and

suggestions are explained.

6.1 Operational control in e-procurement chains

The study was to address the operational control of e-procurement chains through

performance measurement. The operability factor for business continuity was the main

factor behind the operational control. Operability refers to the ability to keep an equipment,

a system or a whole industrial installation in a safe and reliable functioning condition,

according to pre-defined operation requirements.

The chain operability in an e-procurement business consists of individual operability of

systems and chain participants and their collective efforts. The latter due to the fact that in e-

procurement activities there are systems to enable e-procurement and there are parties (e.g.

buyers and sellers) that interact with the system and to each other.

Performance measurement of the chain produces key insights into the level of operability of

the chain.

Therefore the focus of this study was performance measurement of e-procurement chain to

gain operation control over the chain and therefore the following main research question

was formulated:



The ultimate objective of the study was to develop a performance measurement approach for

operational control of e-procurement chains.

6.2 PEPM approach

Analysis of e-procurement and performance measurement studies resulted in a process-

oriented approach that explains how performance of an e-procurement chain is measured. A

logical structure prior to defining PEPM approaches was created. This structure explains

aspects of e-procurement systems and chains that need to be controlled and ultimately

measured. Figure 26 illustrates aspects that are needed to be monitored and controlled.



Operational Control of E-procurement chain


services


services


services


of services


services








Control on quality of service and supporting processes

Figure 26: Constructs for operational control of e-procurement chains

To measure constructs of monitoring in a systematic manner, a process-oriented

performance measurement approach (i.e. PEPM approach) was composed. This method

clearly identified what needs to be controlled and measured and how it has to be measured.

Through application of PEPM approach the management is able to gain insights on the level

of operability of the chain. A total overview of PEPM approach and its steps are illustrated in

figure 27.

The designed method can be then used by organizations that offer e-procurement e-services.

The method clearly outlines the set of services and processes for controlling and measuring.

6.3 PEPM contributions

1. The designed approach can be applied independent of the organizational structures of e-

procurement chain. The process-oriented aspect of the approach causes viewing the

chain as the set of processes. This fact results in applicability of the PEPM approach to

many different forms of e-procurement chains.




its chain



Step 4: Set process




performance metrics


metrics


Figure 27: Total overview of PEPM approach

2. The approach is in a way unique from the applicability point of view. There are many

studies that provide approaches and methods for performance measurement of supply

chain management, e-business, and information systems. Most of these papers

introduced set of performance metrics based on their designed approaches but did not

address how their method can be applied. On the contrary, the PEPM approach is a

practical approach that specifically explains detailed steps toward performance

measurement.

3. Application of approaches such as GQM and BSC for performance measurement of e-

procurement chains heavily relies on discussion rounds, management opinions and

organizational goals. These approaches do not provide guidelines for their implementers

of what is required to be measured for the implementers of the approach. On the

contrary, PEPM approach addresses what is needed to be measured and the implementer

of the approach is guided through the process of performance measurement with clear

steps.

4. With respect to the relation between PEPM approach and ITIL, PEPM gives a

performance measurement view to ITIL. The set of ITIL processes are outlined in order

to achieve different performance measurement goals.

5. Efficiency and applicability objectives were the design issues while developing PEPM

approach. The approach is fast-to-apply because it provides specific steps for

performance measurement. Application of each step is facilitated by the use of current



methods. It is perceived that PEPM approach takes less time to apply than other studied

approaches. PEPM approach is process-oriented thus independent of organizational

structures and provides specific steps that guide the implementer of the approach

through the steps. Therefore PEPM approach is perceived to have high level of

applicability. However, the more definite assessment of these two factors can be obtained

when PEPM is implemented by a third-party implementer.

6. Rabobank Concern Inkoop (RCI) acts as chain management of Raboshop. Monitoring

and controlling the chain and belonging business are the main responsibilities of RCI.

Therefore PEPM approach can be used as a supportive tool for process control and

measurement. It can be applied periodically in order to gain control on processes and

change or update set of performance metrics when changes happen within the business.

6.4 Recommendations

1. A performance measurement system such as GQM-based system introduced by van

Solingen and Berghout (1999) consists of the definition, performance metrics discovery,

data collection, and interpretation phases (figure 13). The scope of the designed PEPM

approach is to discover the performance metrics. The approach does not address how the

performance metrics data can be collected and whether some target values should be

assigned on the defined performance metrics. Hence, the discovered performance

metrics should be further analyzed for the eventual assignment of target values.

Thereafter the exact data collection methods and the method for the analysis of the

results have to be considered.

2. The developed PEPM approach was applied to only one case study, i.e. Raboshop. To

gain more reliable conclusions and assess the efficiency and practicality aspects of the

approach, more case studies are needed as well. It is recommended that the developed

approach should be analyzed through application on other e-procurement situations as

well.

6.5 Future research

1. The scope of the developed performance measurement approach was at the operational

control level. Tactical and strategic levels of control are not discussed in PEPM approach.

A study on these levels of control completes performance measurement on all the

management levels. The optimal goal for performance measurement of e-procurement

chains is to create a complete performance measurement approach for the e-

procurement chains for all management levels.



2. In the PEPM approach the processes were represented by modeling them in the process

modeling language BPMN. Conceptual mapping and data modeling for process

representations are the alternative options to model the processes. These two provide a

more detailed and exact specification of the processes which define and relate the

performance metrics for specific process data. However it is perceived that conceptual

mapping and data modeling are less suitable for management to understand process

steps. A further research on the most suitable data representation method for PEPM

approach is desired.

6.6 Delimitation

1. The set of studied performance measurement approaches are limited to a few whereas

an abundance of approaches existed. However, the studied approaches are widely

appreciated and used in both academic papers and industrial solutions.

2. An evaluation of derived performance metrics is done with respect to the process of

PEPM approach itself. Due to time constraints, gauging of these measures is not carried

out for Raboshop and therefore it is infeasible to determine the appropriateness of

measures to the industry case.



7. References

Angeles, R., 2005. Business -to-business e-procurement corporate initiative: A descriptive

empirical study. Fredericton, NB, Canada, ICEC conference, 2005,

Anthony, R.N., 1965. Planning and Control Systems: A Framework for Analysis. Harvard Business School, Cambridge, MA. Bremser, W.G., Chung, Q.B., 2005. A framework for performance measurement in the e-business

environment. Electronic Commerce Research and Applications 4, 395-412, Elsevier.

Brockers, A., Differding, C., Threin, G., 1996. The Role of Software Process Modeling in

Planning Industrial Measurement Programs. IEEE, Proceedings of Metrics.

Brown, M.G., 2010. Pocket Guide to Performance Management. Quality Press; Spi edition.

Business Performance Improvement Resource (BPIR). URL: http://www.bpir.com. Last

access on July 2010.

COBIT, 2000. Control Objectives. Released by the COBIT Steering Committee and the IT

Governance Institute.

De Boer, L., Harink, J., Heijboer, G., 2001. A Model For Assessing The Impact Of Electronic

Procurement Forms. The 10th International Annual IPSERA Conference.

Dumas, M., O’Sullivan, J., Heravizadeh, M., Edmond, D., ter Hofstede, A., 2001. Towards A Semantic Framework For Service Description. In: Proceedings of the Kluwer Academic Publishers. Folan, P., Browne, J., 2005. A review of performance measurement: Towards performance measurement. Computers in Industry 56, 663-680. Gable, G.G., Sedera, D., Chan, T., 2003. Enterprise Systems Success: A Measurement Model.

Twenty-Fourth International Conference on Information Systems.

Garrett, S.E.G., Skevington, P.J., 1999. An introduction to eCommerce. BT Technol J, Vol 17,

No.3.

Ghandour, A., Deans, K., Benwell, G., 2008. Measuring eCommerce Website Success. 19th

ACIS, Christchurch.

Gunasekaran, A., Patel, C., MacGaughey, R.E., 2004. A framework for supply chain

performance measurement. International journal of production economics 87, 333-347,

Elsevier.



Gunasekaran, A., Patel, C., Tirtiroglu, E., 2001. Performance measures and metrics in a supply

chain environment. International Journal of Operations & Production Management. Vol 21,

No. ½ , pp. 71-87. MCB University Press.

Han, K.H., Kang, J.G., 2007. Two-stage Process Analysis Using the Process-oriented Performance Measurement Framework and Process Simulation. Fifth International Conference on Software Engineering Research, Management and Applications. Information Technology Infrastructure Library (ITIL). URL: http://www.itil.nl. Last accessed

on July 2010.

Jansen-Vullers, M.H., Reijers, H.A., 2003. Business Process Redesign for effective e-commerce processes in the service industry. BETA technical report, Eindhoven University of Technology. Kaplan, R., Norton, D., 1993. Putting the balanced scorecard to work. Harvard Business Review 71 (5), 134-147. Kaplan, R.S., Norton, D.P., 1992. The Balanced Scorecard – Measures that Drive Performance.

Harvard Business Review.

Kim, J., Suh, E., Hwang, H., 2003. A model for evaluating the effectiveness of CRM using the

balanced scorecard. Journal of Interactive Marketing, Volume 17, Number 2.

Klinkenberg, N., Rietveld, E., 2002. De knikkers en het spel – Ondernemerschap voor managers.

Thema.

Ko, R.K.L, 2009. A Computer Scientist’s Introductory Guide To Business Process Management (BPM). ACM, Crossroads, Volume 15, No.4. Kueng, P., 2000. Process performance measurement system: a tool to support process-based organizations. Total Quality Management, Vol. 11, No. 1, 67-85. Kueng, P., Krahn, A.J.W., 1999. Building a Process Performance Measurement System: some

early experiences. Journal of Scientific & Industrial Research, Vol. 58, No. ¾, pp. 149-159.

Kuzic, J., Fisher, J., Scollary, A., 2002. Electronic commerce benefits, challenges and success

factors in the austoralian banking and finance industry. ECIS, Poland.

Li, J., Li, L., 2005. On the Critical Success Factors for B2B E-marketplace. ICEC’05, Xi’an,

China, ACM.

Lucey, T., 2005. Management Information Systems. London: Thomson, 9th edition, ISBN:

978-1-84480-125-8.

Martinsons, M., Davison, R., Tse, D., 1999. The balanced scorecard: a foundation for the

strategic management of information systems. Decision Support Systems 25, 71-88, Elsevier.



Meier, A., Stormer, H., 2009. eBusiness & eCommerce: Managing the Digital Value Chain.

Springer-Verlag, Berling Heidelberg.

Mistry, J. Performance Measurement In the eCommerce Industry. Worcester Polytechnic

Institute. Journal of Business & Economics Research, volume 1, number 11.

Morisawa, T., 2002. Building Performance Measurement Systems with the Balanced Scorecard

Approach. Nomura Research Institute, No.45.

National e-Procumement Project (NEPP), Desktop Guide to e-Procurement. Part 5: What is

an e-Marketplace… and how to get started. www.nepp.org.uk.

Neely, A., Mills, J., Platts, K., Richards, H., Gregory, M., Bourne, M., Kennerley, M., 2000.

Performance measurement system design: developing and testing a process-oriented approach.

International Journal of Operations and Production Management 20 (10), 1131-1145.

Overmeer, M., Palte, Y., Kars, M., Geelof, R., 2008. Ketenregie bij de WIZ-diensten. Divosa

Academie, Universiteit Twente.

Owen, M., Raj, J., 2003. BPMN and Business Process Management. Introduction to the New

Business Process Modeling Standard. Popkin Software.

Reijers, H.A., 2002. Design and Control of Workflow Processes. Business Process Management

for the Service Industry. Eindhoven University Press Facilities.

Reijers, H.A., Mansar, S.L., 2005. Best practices in business process redesign: an overview and qualitative evaluation of successful redesign heuristics. Omega, The International Journal of Management Science, No. 33, pp. 283-306. Romeyn, T., Rietveld, E., 2009. GrexxBoxx. Whitepaper Grexx.

Seel, C., Thomas, O., 2007. Process Performance Measurement for E-Government: A Case Scenario from the German Ministerial Administration. Systemics, Cybernetics and Informatics, Volume 5 – Number 3. Smith, H., Neal, D., Ferrara, L., Hayden, F., 2002. The Emergence of Business Process

Management. CSC Research Services.

Spanos, N. 100 IT Performance Metrics. Solution Consulting, Computer Aid, Inc.

Sriram, V., Stump, R., 2004. Information technology investments in purchasing: an empirical

investigation of communications, relationship and performance outcomes. Omega 32, The

International Journal of Management Science, 41-55.



Stroh, S., Artley, W., 2001. The Performance-oriented Management Handbook, Establishing an

integrated performance measurement system. Performance-oriented Management Special

Interest Group (PBM SIG).

Subramaniam, C., Shaw, M.J., 2002. A study of the value and impact of B2B E-commerce: the

case of web-base procurement. International Journal of Electronic Commerce, Vol. 5, No.4, 19-

40.

The Control Objectives for Information and related Technology (COBIT). URL:

http://www.isaca.org. Last accessed on July 2010.

Turban, E., King, D., Viehland, D., Lee. J., 2005. Electronic Commerce: A Managerial

Perspective. Pearson-Prentice Hall, Upper Saddle River, NJ.

U.S. General Accounting Office (GAO), 2005. Performance measurement and evaluations.

GAO-05-739SP.

Vaidya, K., Callender, G., Sajeev, A.S.M., Gao, J., 2004. Towards a Model for Measuring the

Performance of e-Procurement Initiatives in the Australian Public Sector: A Balanced Scorecard

Approach. AEGC, University of Melbourne.

van der Aalst, W.M.P, ter Hofstede, A.H.M., Weske, M., 2003. Business process management: A survey. In proceedings of the Business Process Management: International Conference (BPM’03). Van der Aalst, W.M.P., Van Hee, K.M., 2002. Workflow management: models, methods, and

systems. MIT Press.

Van Weele, A., Lakemond, N., Claessens, F., 2003. E-Procurement: Feit of Fictie?.

Management & Informatie.

Weele, A.J. van, Veth, A.F.L, 2001. Concurrentiekracht door e-procurement: nu of nooit? Holland Management Review, 2(25), 54-71. White, S.A., 2004. Introduction to BPMN. IBM Corporation.

Yin, R.K., 2003. Case study research – Design and methods. Sage Publications – London.

Zur Muehlen, M., 2001. Workflow-oriented Process Controlling – Or: What You Can Measure You Can Control. In: Layna Fischer (Ed.): Workflow. Handbook. Future Strategies, Lighthouse Point, FL, pp. 51-77.



Appendices

In this chapter the supplementary material for the thesis report is represented.

Appendix A: BPMN

The processes are modeled in the BPMN language. The following figure illustrates the core

set of this modeling language (taken from BPMI.org). The reader is referred to the tutorial of

BPMN (Owen and Raj, 2003) for further the complete understanding of the language.

Appendix B: ITIL processes In the following table set of ITIL processes are defined.



Process Name Process definition

Service Portfolio Management

To decide on a strategy to serve customers, and to develop the service provider's offerings and capabilities.

Financial Management

To manage the service provider's budgeting, accounting and charging requirements.

Service Catalogue Management

To ensure that a Service Catalogue is produced and maintained, containing accurate information on all operational services and those being prepared to be run operationally. Service Catalogue Management provides vital information for all other Service Management processes: Service details, current status and the services' interdependencies.

Service Level Management

To negotiate Service Level Agreements with the customers and to design services in accordance with the agreed service level targets. Service Level Management is also responsible for ensuring that all Operational Level Agreements and Underpinning Contracts are appropriate, and to monitor and report on service levels.

Risk Management To identify, assess and control risks. This includes analyzing the value of assets to the business, identifying threats to those assets, and evaluating how vulnerable each asset is to those threats.

Capacity Management

To ensure that the capacity of IT services and the IT infrastructure is able to deliver the agreed service level targets in a cost effective and timely manner. Capacity Management considers all resources required to deliver the IT service, and plans for short, medium and long term business requirements.

Availability Management

To define, analyze, plan, measure and improve all aspects of the availability of IT services. Availability Management is responsible for ensuring that all IT infrastructure, processes, tools, roles etc are appropriate for the agreed availability targets.

Continuity Management

To manage risks that could seriously impact IT services. ITSCM ensures that the IT service provider can always provide minimum agreed Service Levels, by reducing the risk from disaster events to an acceptable level and planning for the recovery of IT services. ITSCM should be designed to support Business Continuity Management.

Security Management

To ensure the confidentiality, integrity and availability of an organization's information, data and IT services. Information Security Management usually forms part of an organizational approach to security management which has a wider scope than the IT Service Provider.

Compliance Management

To ensure IT services, processes and systems comply with enterprise policies and legal requirements.

Architecture Management

To define a blueprint for the future development of the technological landscape, taking into account the service strategy and newly available technologies.

Supplier Management

To ensure that all contracts with suppliers support the needs of the business, and that all suppliers meet their contractual commitments.

Change Management

To control the lifecycle of all Changes. The primary objective of Change Management is to enable beneficial Changes to be made, with minimum disruption to IT services.

Project Management

To plan and coordinate the resources to deploy a major Release within the predicted cost, time and quality estimates.

Release Management

To plan, schedule and control the movement of releases to test and live environments. The primary goal of Release Management is to ensure that the integrity of the live



environment is protected and that the correct components are released.

Service Validation and Testing

To ensure that deployed Releases and the resulting services meet customer expectations, and to verify that IT operations is able to support the new service.

Application Development and

Customization

To make available applications and systems which provide the required functionality for IT services. This process includes the development and maintenance of custom applications as well as the customization of products from software vendors.

Service Asset and Configuration Management

To maintain information about Configuration Items required to deliver an IT service, including their relationships.

Knowledge Management

To gather, analyze, store and share knowledge and information within an organization. The primary purpose of Knowledge Management is to improve efficiency by reducing the need to rediscover knowledge.

Event Management

To filter and categorize Events and to decide on appropriate actions. Event Management is one of the main activities of Service Operations.

Incident Management

To manage the lifecycle of all Incidents. The primary objective of Incident Management is to return the IT service to users as quickly as possible.

Request Fulfillment

To fulfill Service Requests, which in most cases are minor (standard) Changes (e.g. requests to change a password) or requests for information.

Access Management

To grant authorized users the right to use a service, while preventing access to non-authorized users. The Access Management processes essentially executes policies defined in IT Security Management. Access Management is sometimes also referred to as Rights Management or Identity Management.

Problem Management

To manage the lifecycle of all Problems. The primary objectives of Problem Management are to prevent Incidents from happening, and to minimize the impact of incidents that cannot be prevented. Proactive Problem Management analyses Incident Records, and uses data collected by other IT Service Management processes to identify trends or significant Problems.

IT Operations Management

To monitor and control the IT services and IT infrastructure. The process IT Operations Management executes day-to-day routine tasks related to the operation of infrastructure components and applications. This includes job scheduling, backup and restore activities, print and output management, and routine maintenance.

Service Evaluation To evaluate service quality on a regular basis. This includes identifying areas where the targeted service levels are not reached, and holding regular talks with business to make sure that the agreed service levels are still in line with business needs.

Process Evaluation To evaluate processes on a regular basis. This includes identifying areas where the targeted process metrics are not reached, and holding regular benchmarking, audits, maturity assessments and reviews.

Definition of Improvement

Initiatives

To define specific initiatives aimed at improving services and processes, based on the results of service and process evaluation. The resulting initiatives are either internal initiatives pursued by the service provider on his own behalf, or initiatives which require the customer’s cooperation.

CSI Monitoring To verify if improvement initiatives are proceeding according to plan



Appendix C: ITIL performance indicators

KPI Service Portfolio management Definition

Number of Planned New Services Percentage of new services which are developed, being triggered by Service Portfolio Management

Number of Unplanned New Services Percentage of new services which are developed without being triggered by Service Portfolio Management

Number of Strategic Initiatives Number of strategic initiatives launched from the Service Portfolio Management process

Number of New Customers Number of newly won customers

Number of Lost Customers Number of customers which were lost to competing service providers

KPI Financial Management Definition

Adherence to Budgeting Process Percent of projects using the standard IT budgeting process

Cost-/ Benefit Estimation Percent of project files containing cost-/ benefit estimates

Post Implementation Review Percent of projects where costs and benefits are verified after implementation

Adherence to Approved Budget Percent of IT expenses exceeding the approved budget

Adherence to Project Resources Percent of expenses exceeding the planned budget for a project

Proposals for Cost Optimization Number of proposals by Financial Management for the optimized use of financial resources

KPI Service Level Management Definition

Services covered by SLAs Number of services covered by SLAs

Services covered by OLAs/ UCs Number of Services where SLAs are backed up by corresponding OLAs/ UCs

Monitored SLAs Number of monitored Services/ SLAs, where weak-spots and counter-measures are reported

SLAs under Review Number of Services/ SLAs which are regularly reviewed

Fulfilment of Service Levels Number of Services/ SLAs where the agreed service levels are fulfilled

Number of Service Issues Number of issues in the service provision, which are identified and addressed in an improvement plan



KPI Capacity Management Definition

Incidents due to Capacity Shortages Number of incidents occurring because of insufficient service or component capacity

Exactness of Capacity Forecast Deviation of the predicted capacity development from actual course

Capacity Adjustments Number of adjustments to service and component capacities due to changing demand

Unplanned Capacity Adjustments Number of unplanned increases to service or component capacity as result of capacity bottlenecks

Resolution Time of Capacity Shortage

Resolution time for identified capacity bottlenecks

Capacity Reserves Percentage of capacity reserves at times of normal and maximum demand

Percentage of Capacity Monitoring Percentage of services and infrastructure components under capacity monitoring

KPI Availability Management Definition

Service Availability Availability of IT Services relative to the availability agreed in SLAs and OLAs

Number of Service Interruptions Number of service interruptions

Duration of Service Interruptions Average duration of service interruptions

Availability Monitoring Percentage of services and infrastructure components under availability monitoring

Availability Measures Number of implemented measures with the objective of increasing availability

IT Service Continuity Management Definition

Business Processes with Continuity Agreements

Percentage of business processes which are covered by explicit service continuity targets

Gaps in Disaster Preparation Number of identified gaps in the preparation for disaster events (major threats without any defined counter measures)

Implementation Duration Duration from the identification of of a disaster-related risk to the implementation of a suitable continuity mechanism

Number of Disaster Practices Number of disaster practices actually carried out

Number of Identified Shortcomings during Disaster Practices

Number of identified shortcomings in the preparation for disaster events which are identified during practices.

KPI IT Security Management Definition

Number of Implemented Preventive Measures

Number of preventive security measures which were implemented in response to identified security threats

Implementation Duration Duration from the identification of a security threat to the implementation of a suitable counter measure



Number of Major Security Incidents Number of identified security incidents, classified by severity category

Number of Security-Related Service Downtimes

Number of security incidents causing service interruption or reduced availability

Number of Security Tests Number of security tests and trainings carried out

Number of Identified Shortcomings during Security Tests

Number of identified shortcomings in security mechanisms which were identified during tests

KPI Supplier Management Definition

Number of Agreed UCs Percentage of contracts underpinned by UCs

Number of Contract Reviews Number of conducted contract and supplier reviews

Number of Identified Contract Breaches

Number of contractual obligations which were not fulfilled by suppliers (identified during contract reviews)

KPI Change Management Definition

Number of Major Changes Number of major changes assessed by the CAB (Change Advisory Board)

Number of CAB Meetings Number of CAB (Change Advisory Board) meetings

Time for Change Clearance Average time from registering an RFC with Change Management until Change clearance

Change Acceptance Rate Number of accepted vs. rejected RFCs

Number of Urgent Changes Number of urgent changes assessed by the ECAB (Emergency Change Advisory Board)

KPI Project Management Definition

Number of Projects Number of major release rollouts under the control of Project Management

Percentage of Projects with Project Charters

Percentage of projects which are started with a signed Project Charter in place

Number of Changes to Project Charter

Number of changes to the Project Charter after project start

Adherence to Project Budget Actual vs. planned consumption of financial and personnel resources

Project Delays Actual vs. planned project completion dates

KPI Release and Deployment Management

Definition

Number of Releases Number of releases rolled out into the productive environment, grouped into Major and Minor Releases

Duration of Major Deployments Average duration of major deployments from clearance until completion

Number of Release Backouts Number of releases which had to be reversed

Proportion of Automatic Release Proportion of new releases distributed automatically



Distribution

KPI Service Validation and Testing Definition

Percentage of Failed Release Component Acceptance Tests

Percentage of release components which fail to pass acceptance tests

Number of Identified Errors Number of identified errors during release testing per release

Time for Error Fixing Time until re-submission of fixed release components

Incidents Caused by New Releases Number of Incidents attributable to new releases

Percentage of Failed Service Acceptance Tests

Percentage of Service Acceptance Tests which fail to obtain the customer’s sign-off

KPI Service Asset and Configuration Management

Definition

Verification Frequency Frequency of physical verifications of CMS contents

Verification Duration Average duration for physical verifications of the CMS contents

Effort for CMS Verifications Average work effort for physical verifications of the CMS contents

CMS Coverage Percentage of configuration components for which data is kept in the CMS

Automatic CMS Update Percentage of configuration components for which data in the CMS is updated automatically

Number of CMS Errors Number of occasions when CMS contents are found to be incorrect

KPI Incident Management Definition

Number of Repeated Incidents Number of repeated Incidents, with known resolution methods

Remotely Resolved Incidents Number of incidents resolved remotely by the Service Desk (i.e. without carrying out work at user's location)

Number of Escalations Number of escalations for incidents not resolved in the agreed resolution time

Number of Incidents Number of incidents registered by the Service Desk - per category

Incident Resolution Time Average time for resolving an incident - per category

First Time Resolution Rate Percentage of incidents resolved at the Service Desk during the first call - per category

Resolution within SLA Rate of incidents resolved during solution times agreed in SLA

Incident Resolution Effort Average work effort for resolving incidents per category

KPI Problem Management Definition

Number of Problems Number of Problems registered by Problem Management - per category

Problem Resolution Time Average time for resolving a problem - per category



Number of Incidents per Problem Average number of incidents linked to the same problem before problem identification

Number of Incidents per Known Problem

Average number of incidents linked to the same problem after problem identification

Time until Problem Identification Average time between first occurance of an incident and identification of the underlying root cause

Problem Resolution Effort Average work effort for resolving problems - per category

KPI Service Evaluation Definition

Number of Customer Complaints Number of received customer complaints

Number of Accepted Customer Complaints

Number of received customer complaints which were accepted as justified

Number of Customer Satisfaction Surveys

Number of formal Customer Satisfaction Surveys carried out during the reporting period

Percentage of Returned Questionnaires

Percentage of questionnaires returned, in relation to all questionnaires being sent out

Number of Service Evaluations Number of formal Service Evaluations carried out during the reporting period

Number of Identified Weaknesses Number of weaknesses which were identified during Service Evaluation, to be addressed by improvement initiatives

KPI Process Evaluation Definition

Number of Process Benchmarkings, Maturity Assessments, and Audits

Number of formal Process Benchmarkings, Maturity Assessments, and Audits carried out during the reporting period

Number of Process Evaluations Number of formal Service Evaluations carried out

Number of Identified Weaknesses Number of weaknesses which were identified during Service Evaluation, to be addressed by improvement initiatives

KPI Definition of Improvement Initiatives

Definition

Number of Improvement Initiatives Number of improvement initiatives, resulting from identified weaknesses during Service and Process Evaluation

Number of Completed Improvement Initiatives

Number of improvement initiatives which were completed during the reporting period

Date post:	11-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Master Thesis Performance measurement in E …Master Thesis -7- TU/e 3.4 STEP 1: E-PROCUREMENT CHAIN...

Documents