1

L643: Evaluation of Information Systems

Week 4: January 28, 2008

2

Measurement

Measurement is “the assignment of values to outcomes following a set of rules” E.g., Measurement of tastiness

Good Bad

E.g., measurement of size 2.5 inches 1 inch

3

Conceptualization (Chambliss & Schutt, 2006)

Define the concept (or purpose) e.g., PDA vs. Laptop in a hospital

Identify variables that correspond to the concept e.g., ????

Determine how we can measure these variables [use available data, construct Qs, make observations, content analysis] e.g., ????

4

Scales of Measurement (Salkind, 2007)

Nominal – attributes are named E.g.: nationality (American, Canadian, British)

Ordinal – attributes can be ordered E.g., level of use (Low, High)

Interval – distance is meaningful E.g., temperature in Fahrenheit (11, F, 30 F, 50 F)

Ratio – absolute zero is included E.g., # of clients in 6 months (0, 10, 30)

5

Reliability of Measurement

A reliable measurement is one that gives consistent results Test-retest reliability Parallel forms reliability Internal consistency reliability Interrater reliability

6

To Increase Reliability

Standardized instructions Adequate sample size Avoid unclear items/vague statement Adjust difficulties Minimize external factors

7

Validity of Measurement

Validity = It measures what it is supposed to measure Content validity (by expert) Criterion validity (by a set of existing criteria) Construct validity (by conceptual theory)

8

9

Reliability vs. Validity

10

Population & Sample (Chambliss & Schutt, 2006)

A

B C

Others

A

B C

Others

PopulationSample

11

Sample Size

Large

Small

Small Large

Errors

Size of Sample

12

Sampling

What is a sample? In a research context, “any group on which

information is obtained” The larger group to which one hopes to apply the

results is called the population E.g., All 700 students at a State University who are

majoring in mathematics, constitutes a population E.g., 50 of those students constitute a sample

Random sampling vs. non random sampling

13

Random Sampling

the premise that a sample represents a population is based on the assumption that the sample has been selected at random from the population of interest

14

Definition of An Information System (Davis, 1994)

A set of hardware, software, data, procedural, and human components that work together to generate, collect, store, retrieve, process, analyze, and/or distribute information

The purpose of an IS is to get the right information to the right people at the right time

Davis, W. S. (1994). Business Systems Analysis and Design. Belmont, CA: Wadsworth.

15

Information System Objectives

Why do we need to consider system objectives?

Each system will be designed to meet certain performance objectives

These objectives affect the design of evaluation: They are the evaluation criteria

16

Information System Objectives

Which objectives are mentioned or implied in each of the cases?

John Fluevog Boots & Shoes Zipcar LibraryThing

17

A Taxonomy of Information Systems Success

System Quality Measures of the information processing system itself

Information Quality Measures of information system output , e.g.,

accuracy, meaningfulness, and timeliness Information/System Use

Recipient consumption of the output of an information system

18

A Taxonomy of Information Systems Success

User Attitudes Satisfaction Recipient response to the use of the output of an

information system

Individual Impact The effect of an information system on the behavior of

the recipient

Organizational Impact The effect of the information product on organizational

performance

19

Observations by DeLone & McLean (1992)

There is no consensus on particular measure of information systems success

Simplifying different dependent variable measures would contribute to the MIS research (the results can be compared)

Few field studies to measure organizational performance

6 success categories indicate MIS success is a multidimensional construct & that it should be measured as such (Only 28/100 studied multiple categories)

20

Independent vs. Dependent Variables

Examples of variables Survey format—online, paper-based (IV) Response rates (DV)

IndependentVariable(s)

(presumed or possible cause)

affectsDependent Variable(s)

(presumed results)

21

Measuring the value of information (Ahituv, 1989)

Ahituv, N. (1989). Assessing the value of information: Problems and approaches. Proceedings of the Tenth International Conference on Information Systems, December 4-6, Boston, 315-325.

RealWorld

Data Informationsystem

Decisionmaker

Decisions,actions

Outcomes

Point of measurementperceived value of

information

Point of measurementrealistic value of

information

22

IS Success Model (DeLone & McLean, 1992)

23

I/S Success Model, Expanded(based on Hwang, Windsor, & Pryor, 2000)

Systemquality

Informationquality

Use

Usersatisfaction

Individualimpact

OrganizationalimpactInfoSys

Userenvironment

Organizationalenvironment

0.37

0.47

0.37

0.36

0.43

24

Updated D&M IS Success Model (2002, 2003)

InformationQuality

System Quality

ServiceQuality

IntentionTo Use

Use

UserSatisfaction

NetBenefits

Creation Use Consequences

25

26

Evaluation

The man with the tusk states that an elephant is like a spear. The man with the tail argues that the elephant is like a rope. The man with the trunk says, no, its like a snake. The man with the side thinks its more like a wall. But the man with the leg is sure the elephant is like a tree.

The flaw in all their reasoning is that speculating on the WHOLE from too few FACTS can lead to VERY LARGE errors in judgment.

How does this fable relate to System Evaluation?

27

28

Values in Evaluation (Davidson, 2005)

Suppose that a client does not like the findings of your evaluation and says, “Well, that’s just your opinion about the program. Evaluations are always just so subjective.” How would you respond?

29

Class Activities: Evaluation of IUCAT

1. Write down the assigned category on the worksheet. Individually fill out the worksheet #1 - #3 (p. 1)

2. Work with a group of 5 people to fill out the worksheet for the category (p.2)

3. Present your discussions to the whole class

4. Use the worksheet (p.2) to jot down the main points