Design Science Methodology - Universiteit Twenteroelw/DeutscheTelekom20100902… · Design Science...

Design Science Methodology

Roel WieringaRoel WieringaUniversity of TwenteThe Netherlands

2nd September 2010 Deutsche Telekom 1

What is design science?What is design science?

• Design science is technical science, engineering science• It validates proposed artefacts

– New jet propulsion technology– New information risk assessment method

• And studies implemented artefacts• And studies implemented artefacts – Steam machines– Smallpox vaccinationS a po acc at o– IS impact studies

• Natural science studies entities not constructed by people


OutlineOutline

1. A framework for design science2. A methodology for design science


Economy“Artefactbase”

Decisions about scarce

ISdesign

Goals,Budget Artifacts Goals,

BudgetScientific

knowledgeInstruments,

resources

design science

Artifacts

Budget Budget knowledge

Technology Science

subjects ofstudy

Goals,Budgets

Researchquestion

investigation

Practicalproblemsolving

Technology Science

Scientificknowledge

Scientificknowledge

Problem solving

knowledgeFlow of money,data material;

Knowledgebase

data, material;no control flow


Economy“Artefactbase”

Decisions about scarce

ISdesign

Goals,Budget Artifacts Goals,

BudgetScientific

knowledgeInstruments,

resources

design science

Artifacts

Budget Budget knowledge

Technology ScienceHevner

subjects ofstudy

Goals,Budgets

Researchquestion

investigation

Practicalproblemsolving

Technology ScienceHevner,March,Park,Ram Scientific

knowledge

Ram(2004)

Scientificknowledge

Problem solving

knowledgeFlow of money,data material;

Knowledgebase

data, material;no control flow


Iteration over science and technology• Practical problem: Confidentiality risk assessment in outsourcing

• Artefact proposal: New risk assessment methodTech

Science

• Validation question: Does it work?

• Validation research designs:Science

– Opinion poll– Lab experiment (cases solved by students)– Field experiment (cases solved by professionals)

Action research (researcher uses method in practice and then reflects on– Action research (researcher uses method in practice and then reflects on experience)

– …

B k h i l bl Diff bl d di ?• Back to the practical problem: Different problem understanding?

• Back to the artefact proposal: Improvemenents?Science

• Redo the validation

• ...

Tech

Science


Practical problem relevancePractical problem relevanceG l t b hi d• Goal to be achieved– Normal problems: Goal is stated by stakeholders; limited to what they can imagineto what they can imagine

• Achieve economic goal for the first time• Repair• Improve• Improve

– Radical problems: Goal not stated by stakeholders; limited b i i ti f i / i ti t/ tby imagination of engineer/scientist/entrepeneur

• Circumventing predicted performance limits• Meeting predicted demand

• Relevance is context‐dependent: people, technology, time, place, …


p ,

Research question relevanceResearch question relevance

Questions that come up in an engineering context:– Artifact & Context ~> Effect?

• Will it work?• Why does it fail?• Why does it work?• Why does it work?

– Effect satisfies stakeholder goals?• Will it satisfy goals?

Validation and evaluation questions

• Will it satisfy goals?• Why does it fail to satisfy goals?• Why does it satisfy goals?

– How to measure this?– How to compute this?

Conceptual questions


OutlineOutline

1. A framework for design science– The framework– Sources of relevance

2. A methodology for design science


Practical problems versus knowledge problems

i l bl• Practical problem– Difference between current state of the world and what a stakeholder would like it to be

• To solve it we need to change the world

• Knowledge problem– Difference between what current stakeholder knows and what the stakeholder wants to know

• To solve it stakeholder needs to change their knowledge of the world


What kind of problem?What kind of problem?• What is the architecture of the communication• What is the architecture of the communication infrastructure between A and B?– K Problem: infrastructure exists, stakeholder does not f ,know what its architecture is

Wh t i hit t f• What is an architecture of …– P Problem: A blueprint must be made

Misleading!• Design an architecture for …

– P Problem: A blueprint must be made

Misleading!– P Problem: A blueprint must be made


Heuristics• Practical problems

– Are solved by changing • Knowledge questions

– Are solved by changing the the state of the world

– Solution criterion is ili

knowledge of stakeholders.– Solution criterion is truth

utility• Problem‐dependent: stakeholders and goals

• Problem‐independent: no stakeholders

• One solution; butstakeholders and goals• Many solutions; but trade‐offs

One solution; but approximations

– Technology – Science

Doing ThinkingDoingChanging the worldFuture‐oriented

ThinkingChanging our mindPast‐oriented


OutlineOutline


2. A methodology for design scienceTh i i l– The engineering cycle

– The research cycle


Solving practical problems rationally• The engineering cycle (a.k.a. The regulative cycle)

– problem investigation •K Stakeholders?•K Goals, criteria?•K Phenomena diagnosis?

– treatment design

•K Phenomena, diagnosis?

•P Specify a treatmentA k a sol tion

– design validation •K Will it work?K Will it ti f it i ?

A.k.a. solution

– implementation

•K Will it satisfy criteria?•K Trade‐offs?•K Sensitivity?

Design = deciding what to doSpecification = documenting that decision

implementation

i l t ti l ti K D it k?

Transfer to practice


– implementation evaluation •K Does it work?•K Does it satisfy criteria?

The engineering cycle elaboratedImplementation evaluation = Problem investigationProblem investigation

Implementation Stakeholders? Normal or radical goals?C it i ?Criteria?

Problematic phenomena?Impacts?Diagnosis?

Treatment designDesign validation

Diagnosis?Evaluation?

Treatment design

Expected impact: Context & Treatment → Effects?Expl n ti n?

Normal goals:Satisfy criteriaRepair failuresExplanation?

Evaluation: Effects satisfy goals?Trade-offs for different Solutions?Sensitivity for different Contexts?

Repair failuresImprove performance

Radical goals:Circumvent future failure


Sensitivity for different Contexts? Circumvent future failureSatisfy future demand

Mutual nestingMutual nesting• Doing i i l h i ti l bl f i l ki d• Doing empirical research is a practical problem of a special kind:

– Do something to acquire the knowledge!

• Special kind engineering cycle.– What is the research problem?– How to answer it (research design)?( g )– Is the research design valid?– Do the research– Evaluate the results

• Mutual nestingIn design science the top level cycle is to serve other stakeholders’ goals– In design science, the top level cycle is to serve other stakeholders goals

– In pure science, the top level cycle is to server the researchers’ knowledge goals


OutlineOutline


2. A methodology for design scienceTh i i l– The engineering cycle

– The research cycle


Research cycleAnalysis of resultsAnalysis of results•Observations•Explanation•Generalization

Research probleminvestigation

•Generalization•Practical implications

investigation•Research goal•Problem owner•Unit of study

Researchexecution

•Unit of study•Research questions•Conceptual model•Current knowledgeCurrent knowledge

Research designU it f d t ll ti

Design validationC l i lidit •Unit of data collection

•Environment of data collection•Interaction with unit of data collection•Measurement instruments

•Conclusion validity•Construct validity•Internal validity•External validity


•Measurement instruments•External validity

Research cycleAnalysis of resultsAnalysis of results•Observations•Explanation•Generalization

Research probleminvestigation

•Generalization•Practical implications

investigation•Research goal•Problem owner•Unit of study

Researchexecution

•Unit of study•Research questions•Conceptual model•Current knowledgeCurrent knowledge

Research designU it f d t ll ti

Design validationC l i lidit •Unit of data collection

•Environment of data collection•Interaction with unit of data collection•Measurement instruments

•Conclusion validity•Construct validity•Internal validity•External validity


•Measurement instruments•External validity

Example 1Example 1

• Developing and validating a method for estimating effort of developing process‐aware g p g pinformation systems at Daimler

• See next slide• See next slide– Mutschler, B. (2008) Modeling and simulating causal dependencies on process‐

aware information systems from a cost perspective. PhD thesis, Univ. of Twente. ISBN 978‐90‐365‐2578‐7


How can we improve financial evaluation of process‐aware information systems?

bl ProblemProblemK Current problemswith evaluation?

K Current approaches to

St, Ph, Go, Cr

K Build taxonomy

Problemdecomposition

Problemsequence

financial evaluation? of approaches

K Classify approaches

K Criteria for taxonomies?K Collect taxonomies

K EvaluateK Evaluate them K Validate classification P Design new one

K Validate against criteria

K Evaluate them

P Develop new approach:Causal loop models

K Make causal loop modelsof cost factors of PAIS

K Collect modeling guidelines

P Acquire modelingtools

K V lid t it K Check designK Validate it K Check designargument P Experiment to test one model

P Pilot study using another model


P Pilot study using another modelK Reflection: lessons learned

Validation researchValidation research

• Characteristic of validation research:– The treatment has not been transferred to practice yet

– How to investigate something that does not exist ?How to investigate something that does not exist ?• Mathematical analysis, if possible• Modeling and simulation• Modeling and simulation


Technical validation researchProblem investigation

Treatment design

•Research problem investigationContext & Treatment produce Effects?Effects satisfy goals?

Treatment design

Treatment validation

Trade‐offs?Sensitivity?

•Research design

Treatment implementation

Implementation evaluation

•Research design validationWill this answer our questions?

•Research executionA l i d l i f lImplementation evaluation •Analysis and evaluation of results

Observations?Answers to research questions?Explanation?Explanation?Generalization?Practical impact?

Practical problem Research problem


Technical action researchProblem investigation

Problem classTreatment design

•Research problem investigationContext & Treatment produce Effects?Effects satisfy goals?

Treatment design

Treatment validation

Trade‐offs?Sensitivity?

•Research designProblem investigation

Client’s problemTreatment implementation

Implementation evaluation

•Research design validationWill this answer our questions?

•Research executionA l i d l i f l

Client s problemTreatment design

Customize treatmentTreatment validationImplementation evaluation •Analysis and evaluation of results

Observations?Answers to research questions?Explanation?

Check with clientTreatment implementation

Explanation?Generalization?Practical impact?

Implementation evaluationClient’s goals satisfied?

Practical problem Research problem Practical problem


Example 2Example 2

• Developing a confidentiality risk assessment method when IT is outsourced

• Researcher used method herself to help a client do this assessmentclient do this assessment

• Next slide– (Morali, A. and Wieringa, R.J. (2010) Risk‐Based Confidentiality Requirements

Specification for Outsourced IT Systems. In: Proceedings of the 18th IEEE International Requirements Engineering Conference (RE 2010), 27 Sept ‐ 1 Oct 2010, Sydney, Australia. IEEE Computer Society. )



Validation methods in design scienceCond. of pract.

Cntrl of cntxt

Unit of data collect.

Example User GoalsScaling up toConditions of practice

Illustration no yes model small designer illustration

Opinion imagined yes model any stakeh. support

Lab demo no yes model realistic designer knowledge

Lab expt. no yes model ! artificial subjects knowledge

Benchmark no yes model standard designer knowledge

Field trial yes yes model realistic designer knowledge

Field experiment

yes yes model realistic stakeh. knowledge

Action case yes no model real designer knowledge andAction case yes no model real designer knowledge and change

Pilot project yes no model realistic stakeh. knowledge and change


changeCase study yes no model real stakeh. knowledge and

change

Levels of knowledgeIdealized

N=∞ Universal theories Basic science

Idealizedconditions

k d h i

Di ti th i T t t th i Design science:

Background theories

N=kDiagnostic theories Treatment theories Design science:

Problem classes

Application Generali ation

Context Goal oriented

Application Generalization

N=1

Context X ――→ Effects, should satisfy Goals

Treatment

Goal‐orientedRequirementsEngineering

Trade-offs (effectiveness of other treatments?)Sensitivity (robust under future scenarios?)Conditions

of practice


Take home1. In design science research, we iterate over technology (practical

problem solving) and research (knowledge question answering)

2. Much of design science research is validation research– Treatment & Context produces effects?– Treatment & Context produces effects?– Effects satisfy criteria?– Trade‐offs?

S iti it ?– Sensitivity?

3. Validation research must simulate the treatment in practice– Opinion poll of practitioners– Experimenting with models– Action case


DiscussionDiscussion


Date post:	22-Jun-2020
Category:	Documents
Upload:	others
View:	9 times
Download:	1 times

Design Science Methodology - Universiteit Twenteroelw/DeutscheTelekom20100902… · Design Science...

Documents