Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | noreen-alice-jackson |
View: | 217 times |
Download: | 2 times |
1
L643: Evaluation of Information Systems
Week 4: January 28, 2008
2
Measurement
Measurement is “the assignment of values to outcomes following a set of rules” E.g., Measurement of tastiness
Good Bad
E.g., measurement of size 2.5 inches 1 inch
3
Conceptualization (Chambliss & Schutt, 2006)
Define the concept (or purpose) e.g., PDA vs. Laptop in a hospital
Identify variables that correspond to the concept e.g., ????
Determine how we can measure these variables [use available data, construct Qs, make observations, content analysis] e.g., ????
4
Scales of Measurement (Salkind, 2007)
Nominal – attributes are named E.g.: nationality (American, Canadian, British)
Ordinal – attributes can be ordered E.g., level of use (Low, High)
Interval – distance is meaningful E.g., temperature in Fahrenheit (11, F, 30 F, 50 F)
Ratio – absolute zero is included E.g., # of clients in 6 months (0, 10, 30)
5
Reliability of Measurement
A reliable measurement is one that gives consistent results Test-retest reliability Parallel forms reliability Internal consistency reliability Interrater reliability
6
To Increase Reliability
Standardized instructions Adequate sample size Avoid unclear items/vague statement Adjust difficulties Minimize external factors
7
Validity of Measurement
Validity = It measures what it is supposed to measure Content validity (by expert) Criterion validity (by a set of existing criteria) Construct validity (by conceptual theory)
8
9
Reliability vs. Validity
10
Population & Sample (Chambliss & Schutt, 2006)
A
B C
Others
A
B C
Others
PopulationSample
11
Sample Size
Large
Small
Small Large
Errors
Size of Sample
12
Sampling
What is a sample? In a research context, “any group on which
information is obtained” The larger group to which one hopes to apply the
results is called the population E.g., All 700 students at a State University who are
majoring in mathematics, constitutes a population E.g., 50 of those students constitute a sample
Random sampling vs. non random sampling
13
Random Sampling
the premise that a sample represents a population is based on the assumption that the sample has been selected at random from the population of interest
14
Definition of An Information System (Davis, 1994)
A set of hardware, software, data, procedural, and human components that work together to generate, collect, store, retrieve, process, analyze, and/or distribute information
The purpose of an IS is to get the right information to the right people at the right time
Davis, W. S. (1994). Business Systems Analysis and Design. Belmont, CA: Wadsworth.
15
Information System Objectives
Why do we need to consider system objectives?
Each system will be designed to meet certain performance objectives
These objectives affect the design of evaluation: They are the evaluation criteria
16
Information System Objectives
Which objectives are mentioned or implied in each of the cases?
John Fluevog Boots & Shoes Zipcar LibraryThing
17
A Taxonomy of Information Systems Success
System Quality Measures of the information processing system itself
Information Quality Measures of information system output , e.g.,
accuracy, meaningfulness, and timeliness Information/System Use
Recipient consumption of the output of an information system
18
A Taxonomy of Information Systems Success
User Attitudes Satisfaction Recipient response to the use of the output of an
information system
Individual Impact The effect of an information system on the behavior of
the recipient
Organizational Impact The effect of the information product on organizational
performance
19
Observations by DeLone & McLean (1992)
There is no consensus on particular measure of information systems success
Simplifying different dependent variable measures would contribute to the MIS research (the results can be compared)
Few field studies to measure organizational performance
6 success categories indicate MIS success is a multidimensional construct & that it should be measured as such (Only 28/100 studied multiple categories)
20
Independent vs. Dependent Variables
Examples of variables Survey format—online, paper-based (IV) Response rates (DV)
IndependentVariable(s)
(presumed or possible cause)
affectsDependent Variable(s)
(presumed results)
21
Measuring the value of information (Ahituv, 1989)
Ahituv, N. (1989). Assessing the value of information: Problems and approaches. Proceedings of the Tenth International Conference on Information Systems, December 4-6, Boston, 315-325.
RealWorld
Data Informationsystem
Decisionmaker
Decisions,actions
Outcomes
Point of measurementperceived value of
information
Point of measurementrealistic value of
information
22
IS Success Model (DeLone & McLean, 1992)
23
I/S Success Model, Expanded(based on Hwang, Windsor, & Pryor, 2000)
Systemquality
Informationquality
Use
Usersatisfaction
Individualimpact
OrganizationalimpactInfoSys
Userenvironment
Organizationalenvironment
0.37
0.47
0.37
0.36
0.43
24
Updated D&M IS Success Model (2002, 2003)
InformationQuality
System Quality
ServiceQuality
IntentionTo Use
Use
UserSatisfaction
NetBenefits
Creation Use Consequences
25
26
Evaluation
The man with the tusk states that an elephant is like a spear. The man with the tail argues that the elephant is like a rope. The man with the trunk says, no, its like a snake. The man with the side thinks its more like a wall. But the man with the leg is sure the elephant is like a tree.
The flaw in all their reasoning is that speculating on the WHOLE from too few FACTS can lead to VERY LARGE errors in judgment.
How does this fable relate to System Evaluation?
27
28
Values in Evaluation (Davidson, 2005)
Suppose that a client does not like the findings of your evaluation and says, “Well, that’s just your opinion about the program. Evaluations are always just so subjective.” How would you respond?
29
Class Activities: Evaluation of IUCAT
1. Write down the assigned category on the worksheet. Individually fill out the worksheet #1 - #3 (p. 1)
2. Work with a group of 5 people to fill out the worksheet for the category (p.2)
3. Present your discussions to the whole class
4. Use the worksheet (p.2) to jot down the main points