Research MethodologyResearch Methodology

BY: Jocelyn M. GallegosBY: Jocelyn M. Gallegos

Benefits of research to whomBenefits of research to whom

As a graduate student...As a graduate student... To be able to read and understand the To be able to read and understand the

empirical literature in your field; to become a empirical literature in your field; to become a critical consumer of information.critical consumer of information.

As a graduate student preparing for a As a graduate student preparing for a thesis or dissertation…thesis or dissertation… To be able to both design and implement your To be able to both design and implement your

thesis or dissertation as well as future studies thesis or dissertation as well as future studies that interest you.that interest you.

Benefits to whomBenefits to whom

As a future practitioner…As a future practitioner… To be able to intelligently participate in To be able to intelligently participate in

research projects, evaluations, and studies research projects, evaluations, and studies undertaken by your institution.undertaken by your institution.

As an educated citizen ...As an educated citizen ... To understand the difference between To understand the difference between

scientifically acquired knowledge and other scientifically acquired knowledge and other kinds of information.kinds of information.

What Research Is NotWhat Research Is Not

Research isn’t information gathering:Research isn’t information gathering: Gathering information from resources such Gathering information from resources such

books or magazines isn’t research.books or magazines isn’t research. No contribution to new knowledge.No contribution to new knowledge.

Research isn’t the transportation of facts:Research isn’t the transportation of facts: Merely transporting facts from one resource to Merely transporting facts from one resource to

another doesn’t constitute research.another doesn’t constitute research. No contribution to new knowledge although No contribution to new knowledge although

this might make existing knowledge more this might make existing knowledge more accessible.accessible.

What Research IsWhat Research Is

Research is:Research is:

“…the systematic process of collecting and analyzing information (data) in order to increase our understanding of the phenomenon about which we are concerned or interested.”1

What’s the Difference Between What’s the Difference Between “Method” and “Methodology”?“Method” and “Methodology”?

Method:Method: Techniques for Techniques for

gathering evidencegathering evidence The various ways The various ways

of proceeding in of proceeding in gathering gathering informationinformation

Methodology:Methodology: The underlying The underlying

theory and analysis theory and analysis of how research of how research does or should does or should proceed, often proceed, often influenced by influenced by disciplinediscipline

Epistemology, Methodology, and Epistemology, Methodology, and MethodMethod

““a research a research methodmethod is a technique is a technique for (or way of proceeding in) for (or way of proceeding in) gathering evidence" whilegathering evidence" while

""methodologymethodology is a theory and is a theory and analysis of how research does or analysis of how research does or should proceed" andshould proceed" and

"an "an epistemologyepistemology is a theory of is a theory of knowledge"knowledge"

"It is the theory that decides "It is the theory that decides what can be observed."what can be observed."

Albert EinsteinAlbert Einstein

Research CharacteristicsResearch Characteristics1.1. Originates with a question or problem.Originates with a question or problem.

2.2. Requires clear articulation of a goal.Requires clear articulation of a goal.

3.3. Follows a specific plan or procedure.Follows a specific plan or procedure.

4.4. Often divides main problem into subproblems.Often divides main problem into subproblems.

5.5. Guided by specific problem, question, or Guided by specific problem, question, or hypothesis.hypothesis.

6.6. Accepts certain critical assumptions.Accepts certain critical assumptions.

7.7. Requires collection and interpretation of data.Requires collection and interpretation of data.

8.8. Cyclical (helical) in nature.Cyclical (helical) in nature.

Research ProjectsResearch Projects

Research begins with a problem.Research begins with a problem. This problem need not be Earth-shaking.This problem need not be Earth-shaking.

Identifying this problem can actually be Identifying this problem can actually be the hardest part of research.the hardest part of research.

In general, good research projects should:In general, good research projects should: Address an important question.Address an important question. Advance knowledge.Advance knowledge.

Research Project PitfallsResearch Project Pitfalls

The following kinds of projects usually The following kinds of projects usually don’t make for good research:don’t make for good research: Self-enlightenment.Self-enlightenment. Comparing data sets.Comparing data sets. Correlating data sets.Correlating data sets. Problems with yes / no answers.Problems with yes / no answers.

High-Quality ResearchHigh-Quality Research(1 of 2)(1 of 2)

Good research requires:Good research requires: The scope and limitations of the work to be The scope and limitations of the work to be

clearly defined.clearly defined.

The process to be clearly explained so that it The process to be clearly explained so that it can be reproduced and verified by other can be reproduced and verified by other researchers.researchers.

A thoroughly planned design that is as A thoroughly planned design that is as objective as possible.objective as possible.

High-Quality ResearchHigh-Quality Research(2 of 2)(2 of 2)

Good research requires:Good research requires: Highly ethical standards be applied.Highly ethical standards be applied.

All limitations be documented.All limitations be documented.

Data be adequately analyzed and explained.Data be adequately analyzed and explained.

All findings be presented unambiguously and All findings be presented unambiguously and all conclusions be justified by sufficient all conclusions be justified by sufficient evidence.evidence.

Sources of Research ProblemsSources of Research Problems

Observation.Observation. Literature reviews.Literature reviews. Professional conferences.Professional conferences. Experts.Experts.

Stating the Research ProblemStating the Research Problem

Once you’ve identified a research Once you’ve identified a research problem:problem: State that problem clearly and completely.State that problem clearly and completely. Determine the feasibility of the research.Determine the feasibility of the research.

Identify subproblems:Identify subproblems: Completely researchable units.Completely researchable units. Small in number.Small in number. Add up to the total problem.Add up to the total problem. Must be clearly tied to the interpretation of the Must be clearly tied to the interpretation of the

data.data.

ResearchResearch

Acquisition of KnowledgeAcquisition of Knowledge

Knowledge Knowledge

v/sv/s

InformationInformation

(Theoretically, concerned with developing, (Theoretically, concerned with developing, exploring, or testing theories)exploring, or testing theories)

TheoryTheory

What exists?What exists? Why exists?Why exists?

What will happen in future?What will happen in future?

How to acquire Knowledge?How to acquire Knowledge?

Inductive ReasoningInductive Reasoning( works moving from specific ( works moving from specific

observation to broader observation to broader generalisation, bottom generalisation, bottom approach)approach)

Deductive ReasoningDeductive Reasoning( more general to more specific ( more general to more specific

or top down approach)or top down approach)

Deductive ReasoningDeductive Reasoning

Theory

Hypothesis

Observation

Confirmation

Inductive ReasoningInductive Reasoning

ObservatioObservationn

Pattern

Hypothesis

Theory

PositivismPositivism

Goal of Knowledge is to Goal of Knowledge is to describe the phenomena that describe the phenomena that are experienced,are experienced,

There is interdependence of There is interdependence of observation and theory, our observation and theory, our observations are theory ladenobservations are theory laden

Scientific Scientific thoughtthought

Francis BaconFrancis Bacon Rene DescartesRene Descartes John StuartJohn Stuart Karl PopperKarl Popper Thomas KuhnThomas Kuhn Feyer bendFeyer bend Steven HagenSteven Hagen

HypothesesHypotheses

Hypotheses are tentative, intelligent Hypotheses are tentative, intelligent guesses as to the solution of the problem.guesses as to the solution of the problem. There is often a 1-1 correspondence between a There is often a 1-1 correspondence between a

subproblem and a hypothesis.subproblem and a hypothesis. Hypotheses can direct later research activities Hypotheses can direct later research activities

since they can help determine the nature of since they can help determine the nature of the research and methods applied.the research and methods applied.

DelimitationsDelimitations

All research has limitations and thus All research has limitations and thus certain work that certain work that willwill notnot be performed. be performed.

The work that will not be undertaken is The work that will not be undertaken is described as the described as the delimitationsdelimitations of the of the research.research.

DefinitionsDefinitions

Define each technical term as it is used in Define each technical term as it is used in relation to your research project.relation to your research project. This helps remove significant ambiguity from This helps remove significant ambiguity from

the research itself by ensuring that reviewers, the research itself by ensuring that reviewers, while they may not agree with your definitions, while they may not agree with your definitions, at least know what you’re talking about.at least know what you’re talking about.

AssumptionsAssumptions

Assumptions are those things that the Assumptions are those things that the researcher is taking for granted.researcher is taking for granted. For example: a given test instrument accurately For example: a given test instrument accurately

and consistently measures the phenomenon in and consistently measures the phenomenon in question.question.

As a general rule you’re better off As a general rule you’re better off documenting an assumption than ignoring documenting an assumption than ignoring it.it. Overlooked assumptions provide a prime source Overlooked assumptions provide a prime source

of debate about a research project’s results.of debate about a research project’s results.

Importance of the StudyImportance of the Study

Many research problems have a kind of Many research problems have a kind of theoretical feel about them. Such projects theoretical feel about them. Such projects often need to be justified:often need to be justified: What is the research project’s practical value?What is the research project’s practical value?

Without this justification, it will prove Without this justification, it will prove difficult to convince others that the difficult to convince others that the problem in question is worth study.problem in question is worth study.

Research ProposalsResearch Proposals

Research proposals are documents that Research proposals are documents that describe the intended research including:describe the intended research including: Problem and subproblems.Problem and subproblems. Hypotheses.Hypotheses. Delimitations.Delimitations. Definitions.Definitions. Assumptions.Assumptions. Importance.Importance. Literature review.Literature review.

Literature ReviewLiterature Review

A literature review is a necessity.A literature review is a necessity. Without this step, you won’t know if your Without this step, you won’t know if your

problem has been solved or what related problem has been solved or what related research is already underway.research is already underway.

When performing the review:When performing the review: Start searching professional journals.Start searching professional journals. Begin with the most recent articles you can find.Begin with the most recent articles you can find. Keep track of relevant articles in a bibliography.Keep track of relevant articles in a bibliography. Don’t be discouraged if work on the topic is Don’t be discouraged if work on the topic is

already underway.already underway.

Literature Review PitfallsLiterature Review Pitfalls(1 of 2)(1 of 2)

Be very careful to check your sources Be very careful to check your sources when doing your literature review.when doing your literature review.

Many trade magazines are not peer Many trade magazines are not peer reviewed.reviewed. Professional conferences and journals often Professional conferences and journals often

have each article reviewed by multiple people have each article reviewed by multiple people before it is even recommended for publication.before it is even recommended for publication.

The IEEE and ACM digital libraries are good The IEEE and ACM digital libraries are good places to start looking for legitimate research.places to start looking for legitimate research.

Literature Review PitfallsLiterature Review Pitfalls(2 of 2)(2 of 2)

The Internet can be a good source of The Internet can be a good source of information. It is also full of pseudo-information. It is also full of pseudo-science and poor research.science and poor research.

Make sure you verify the claims of any Make sure you verify the claims of any documentation that has not been peer documentation that has not been peer reviewed by other professionals in the reviewed by other professionals in the computing industry.computing industry.

Processes & MethodologiesProcesses & Methodologies

Research Process.Research Process.

Common Methodologies.Common Methodologies.

Methodology Comparison.Methodology Comparison.

Research ProcessResearch Process

Research is an extremely cyclic process.Research is an extremely cyclic process. Later stages might necessitate a review of Later stages might necessitate a review of

earlier work.earlier work.

This isn’t a weakness of the process but is This isn’t a weakness of the process but is part of the built-in error correction part of the built-in error correction machinery.machinery.

Because of the cyclic nature of research, it Because of the cyclic nature of research, it can be difficult to determine where to start can be difficult to determine where to start and when to stop.and when to stop.

Step 1: A Question Is RaisedStep 1: A Question Is Raised

A question occurs to or is posed to the A question occurs to or is posed to the researcher for which that researcher has researcher for which that researcher has no answer.no answer. This doesn’t mean that someone else doesn’t This doesn’t mean that someone else doesn’t

already have an answer.already have an answer.

The question needs to be converted to an The question needs to be converted to an appropriate problem statement like that appropriate problem statement like that documented in a research proposal.documented in a research proposal.

Step 2: Suggest HypothesesStep 2: Suggest Hypotheses

The researcher generates intermediate The researcher generates intermediate hypotheses to describe a solution to the hypotheses to describe a solution to the problem.problem.

This is at best a temporary solution since there This is at best a temporary solution since there is as yet no evidence to support either the is as yet no evidence to support either the acceptance or rejection of these hypotheses.acceptance or rejection of these hypotheses.

Step 3: Literature ReviewStep 3: Literature Review

The available literature is reviewed to The available literature is reviewed to determine if there is already a solution to determine if there is already a solution to the problem.the problem. Existing solutions do not always explain new Existing solutions do not always explain new

observations.observations. The existing solution might require some The existing solution might require some

revision or even be discarded.revision or even be discarded.

Step 4: Literature EvaluationStep 4: Literature Evaluation

It’s possible that the literature review has It’s possible that the literature review has yielded a solution to the proposed yielded a solution to the proposed problem.problem. This means that you haven’t really done This means that you haven’t really done

research.research.

On the other hand, if the literature review On the other hand, if the literature review turns up nothing, then additional research turns up nothing, then additional research activities are justified.activities are justified.

Step 5: Acquire DataStep 5: Acquire Data

The researcher now begins to gather data The researcher now begins to gather data relating to the research problem.relating to the research problem. The means of data acquisition will often The means of data acquisition will often

change based on the type of the research change based on the type of the research problem.problem.

This might entail only data gathering, but it This might entail only data gathering, but it could also require the creation of new could also require the creation of new measurement instruments.measurement instruments.

Step 6: Data AnalysisStep 6: Data Analysis

The data that were gathered in the The data that were gathered in the previous step are analyzed as a first step previous step are analyzed as a first step in ascertaining their meaning.in ascertaining their meaning.

As before, the analysis of the data does As before, the analysis of the data does not constitute research.not constitute research. This is basic number crunching.This is basic number crunching.

Step 7: Data InterpretationStep 7: Data Interpretation

The researcher interprets the newly The researcher interprets the newly analyzed data and suggests a conclusion.analyzed data and suggests a conclusion. This can be difficult.This can be difficult. Keep in mind that data analysis that suggests Keep in mind that data analysis that suggests

a correlation between two variables can’t a correlation between two variables can’t automatically be interpreted as suggesting automatically be interpreted as suggesting causality between those variables.causality between those variables.

Step 8: Hypothesis SupportStep 8: Hypothesis Support

The data will either support the The data will either support the hypotheses or they won’t.hypotheses or they won’t. This may lead the researcher to cycle back to This may lead the researcher to cycle back to

an earlier step in the process and begin again an earlier step in the process and begin again with a new hypothesis.with a new hypothesis.

This is one of the self-correcting mechanisms This is one of the self-correcting mechanisms associated with the scientific method.associated with the scientific method.

Common MethodologiesCommon Methodologies

Methodologies are high-level approaches Methodologies are high-level approaches to conducting research.to conducting research. The individual steps within the methodology The individual steps within the methodology

might vary based on the research being might vary based on the research being performed.performed.

Two commonly used research Two commonly used research methodologies: methodologies: Quantitative.Quantitative. Qualitative.Qualitative.

Methodology ComparisonMethodology Comparison

QuantitativeQuantitative

Explanation, predictionExplanation, prediction Test theoriesTest theories Known variablesKnown variables Large sampleLarge sample Standardized instrumentsStandardized instruments DeductiveDeductive

QualitativeQualitative

Explanation, descriptionExplanation, description Build theoriesBuild theories Unknown variablesUnknown variables Small sampleSmall sample Observations, interviewsObservations, interviews InductiveInductive

An Overview of Empirical An Overview of Empirical Research MethodsResearch Methods

Descriptive (Qualitative)Descriptive (Qualitative) EthnographyEthnography Case StudyCase Study Suvey/SamplingSuvey/Sampling Focus GroupsFocus Groups Discourse/Text AnalysisDiscourse/Text Analysis Quantitative DescriptionQuantitative Description Prediction/ClassificationPrediction/Classification

Experimental Experimental (Quantitative)(Quantitative)

True ExperimentTrue Experiment Quasi-ExperimentQuasi-Experiment Meta-AnalysisMeta-Analysis

Assessing MethodsAssessing Methods

Research Question(s) is/are keyResearch Question(s) is/are key Methods must answer the research Methods must answer the research

question(s)question(s) Methodology guides applicationMethodology guides application Epistemology guides analysisEpistemology guides analysis

EthnographiesEthnographies

+ Observational field work done in the Observational field work done in the actual context being studiedactual context being studied

+ Focus on how individuals interrelate Focus on how individuals interrelate in their own environment (and the in their own environment (and the influence of this environment)influence of this environment)

- Difficult to interpret/analyzeDifficult to interpret/analyze- Time consuming/expensiveTime consuming/expensive- Can influence subject behaviorCan influence subject behavior

Case StudiesCase Studies

+ Focus is on individual or small groupFocus is on individual or small group+ Able to conduct a comprehensive Able to conduct a comprehensive

analysis from a comparison of casesanalysis from a comparison of cases+ Allows for identification of variables Allows for identification of variables

or phenomenon to be studiedor phenomenon to be studied- Time consumingTime consuming- Depth rather than breadthDepth rather than breadth- Not necessarily representativeNot necessarily representative

Survey ResearchSurvey Research

+ An efficient means of gathering large An efficient means of gathering large amounts of dataamounts of data

+ Can be anonymous and inexpensiveCan be anonymous and inexpensive- Feedback often incompleteFeedback often incomplete- Wording of instrument can bias Wording of instrument can bias

feedbackfeedback- Details often left outDetails often left out

Focus GroupsFocus Groups

+ Aid in understanding audience, group, usersAid in understanding audience, group, users+ Small group interaction more than Small group interaction more than

individual responseindividual response+ Helps identify and fill gaps in current Helps identify and fill gaps in current

knowledge re: perceptions, attitudes, knowledge re: perceptions, attitudes, feelings, etc.feelings, etc.

- Does not give statisticsDoes not give statistics- Marketing tools seen as “suspect”Marketing tools seen as “suspect”- Analysis subjectiveAnalysis subjective

Discourse/Text AnalysisDiscourse/Text Analysis

+ Examines actual discourse produced Examines actual discourse produced for a particular purpose (job, school)for a particular purpose (job, school)

+ Helps in understanding of context, Helps in understanding of context, production, audience, and textproduction, audience, and text

+ Schedule for analysis not demandingSchedule for analysis not demanding- Labor intensiveLabor intensive- Categories often fluid, making Categories often fluid, making

analysis difficultanalysis difficult

Quantitative Descriptive StudiesQuantitative Descriptive Studies

+ Isolates systematically the most Isolates systematically the most important variables (often from case important variables (often from case studies) and to quantify and interrelate studies) and to quantify and interrelate them (often via survey or questionnaire)them (often via survey or questionnaire)

+ Possible to collect large amounts of dataPossible to collect large amounts of data+ Not as disruptiveNot as disruptive+ Biases not as likelyBiases not as likely- Data restricted to information availableData restricted to information available

Discourse/Text AnalysisDiscourse/Text Analysis

+ Examines actual discourse produced Examines actual discourse produced for a particular purpose (job, school)for a particular purpose (job, school)

+ Helps in understanding of context, Helps in understanding of context, production, audience, and textproduction, audience, and text

+ Schedule for analysis not demandingSchedule for analysis not demanding- Labor intensiveLabor intensive- Categories often fluid, making Categories often fluid, making

analysis difficultanalysis difficult

Quantitative Descriptive StudiesQuantitative Descriptive Studies

+ Isolates systematically the most Isolates systematically the most important variables (often from case important variables (often from case studies) and to quantify and interrelate studies) and to quantify and interrelate them (often via survey or questionnaire)them (often via survey or questionnaire)

+ Possible to collect large amounts of dataPossible to collect large amounts of data+ Not as disruptiveNot as disruptive+ Biases not as likelyBiases not as likely- Data restricted to information availableData restricted to information available

Prediction and Classification Prediction and Classification StudiesStudies

Goal is to predict behaviors:Goal is to predict behaviors: PredictionPrediction forecasts and forecasts and intervalinterval variable variable

(Diagnostic/TAAS scores)(Diagnostic/TAAS scores) ClassificationClassification forecasts a forecasts a nominalnominal variable (Major variable (Major

selection after taking 2311)selection after taking 2311)+ Important in industry, education to predict behaviorsImportant in industry, education to predict behaviors- Need substantial populationNeed substantial population- Restricted range of variables can cause Restricted range of variables can cause

misinterpretationmisinterpretation- Variables cannot be added together; must be Variables cannot be added together; must be

weighted and looked at in context of other variablesweighted and looked at in context of other variables

Positive Aspects of Positive Aspects of Descriptive/Qualitative Descriptive/Qualitative

ResearchResearch Naturalistic; allows for subjects to Naturalistic; allows for subjects to

interact with environmentinteract with environment Can use statistical analysisCan use statistical analysis Seeks to further develop theory (not Seeks to further develop theory (not

to influence action); to influence action); PrescientificPrescientific Coding schemes often arise from Coding schemes often arise from

interplay between data and interplay between data and researcher’s knowledge of theoryresearcher’s knowledge of theory

Problems with Problems with Descriptive/Qualitative Descriptive/Qualitative

ResearchResearch Impossible to overlay structureImpossible to overlay structure Impossible to impose controlImpossible to impose control Subject pool often limited, not Subject pool often limited, not

representativerepresentative Seen as more “subjective,” less Seen as more “subjective,” less

rigorousrigorous Beneficial only in terms of initial Beneficial only in terms of initial

investigation to form hypothesisinvestigation to form hypothesis

Experimental Research: True Experimental Research: True ExperimentExperiment

+ Random sampling, or selection, of subjects Random sampling, or selection, of subjects (which are also stratified)(which are also stratified)

+ Introduction of a treatmentIntroduction of a treatment+ Use of a control group for comparing Use of a control group for comparing

subjects who don’t receive treatment with subjects who don’t receive treatment with those who dothose who do

- Adherence to scientific method (seen as Adherence to scientific method (seen as positive, too)positive, too)

- Must have both internal and external validityMust have both internal and external validity- Treatment and control might seem artificialTreatment and control might seem artificial

Experimental Research: Quasi-Experimental Research: Quasi-ExperimentExperiment

+ Similar to Experiment, except that the subjects Similar to Experiment, except that the subjects are not randomized. Intact groups are often are not randomized. Intact groups are often used (for example, students in a classroom). used (for example, students in a classroom).

+ To draw more fully on the power of the To draw more fully on the power of the experimental method, a pretest may be experimental method, a pretest may be employed.employed.

+ Employ treatment, control, and scientific Employ treatment, control, and scientific methodmethod

- Act of control and treatment makes situation Act of control and treatment makes situation artificialartificial

- Small subject poolsSmall subject pools

Meta-AnalysisMeta-Analysis

+ Takes the results of true and quasi-Takes the results of true and quasi-experiments and identifies experiments and identifies interrelationships of conclusionsinterrelationships of conclusions

+ SystematicSystematic+ ReplicableReplicable+ Summarizes overall resultsSummarizes overall results- C/C apples and oranges?C/C apples and oranges?- Quality of studies used?Quality of studies used?

Positive Aspects of Positive Aspects of Experimental ResearchExperimental Research

Tests the validity of generalizationsTests the validity of generalizations Seen as rigorousSeen as rigorous Identifies a cause-and-effect Identifies a cause-and-effect

relationshiprelationship Seen as more objective, less Seen as more objective, less

subjectivesubjective Can be predictiveCan be predictive

Problems with Experimental Problems with Experimental ResearchResearch

Generalizations need to be qualified Generalizations need to be qualified according to limitation of research according to limitation of research methods employedmethods employed

Controlled settings don’t mirror Controlled settings don’t mirror actual conditions; unnaturalactual conditions; unnatural

Difficult to isolate a single variableDifficult to isolate a single variable Doesn’t allow for self-reflection (built-Doesn’t allow for self-reflection (built-

in)in)

Testing the WatersTesting the Waters

How do you come up with a good How do you come up with a good research question?research question?

How do you determine if the method How do you determine if the method you plan to use will answer your you plan to use will answer your question?question?

What epistemology should you use to What epistemology should you use to analyze data?analyze data?

Quantitative Methods Quantitative Methods

SamplingstSamplingst Testing of Hypothesis Testing of Hypothesis Chi Square TestChi Square Test ANOVAANOVA Multivariate AnalysisMultivariate Analysis

ThanksThanks