52
Rigor in academic research Dan Remenyi PhD
Rigor in academic researchDan Remenyi PhD
April 22, 2023
2
April 22, 2023
3
We are in the age of systematic research•Miracles or unexplained revelations are
simply not allow in academic research•Archimedes, Galileo and Newton would
not get a degree for at least some of their discovers
•Rigor is a concept which spans a spectrum or perhaps a continuum. Research needs to be “rigorous enough”.
April 22, 2023
4
A definition of rigor•A piece of research can be said to be rigorous if
there is no doubt that it has been conducted in terms of all the rules associated with the research paradigm under which it was produced.
• The degree to which research methods are scrupulously and meticulously applied
•Is there a trade off between Relevance and Rigor? Some researchers suggest that there is an Heisenberg type relationship but this does not have to be the case.
April 22, 2023
5
Another definition of rigor• Academic research has to end up as an argument that
something of value has been added to the body of theoretical knowledge. Rigor is a question of the force of the argument i.e. how persuasive is this argument. The force of the argument is a function of an appropriate research question/s, useful data, and analysis thereof and a logical/rational and clear argument.
• Some researchers argue that “rigor is the strength of inference made possible by the given research study”. Staw, B. M. (1985) Reports on the road to relevance and rigor: Some unexplored issues in publishing organizational research. In L. L. Cummings & P. J. Frost, eds. Publishing in the Organizational Sciences (pp 96-107). Homewood Illinois, Richard D. Irwin, Inc.
April 22, 2023
6
Definitions of Rationality ….1• A principle aim of this chapter has been to build the
foundation upon which a clear understanding of the concept of “rationality” could be erected. Clarity does not necessarily imply simplicity, however. Roughly speaking, rationality is concerned with the selection of preferred behaviour alternatives in terms of some system of values whereby the consequences of behaviour can be evaluated. Does this mean that the process of adaption must be conscious, or are unconscious processes included as well? It has been shown that many of the steps in mathematical invention – than which there can presumably be nothing more rational – are subconscious; and this is certainly true of the simpler processes of equation-solving.
April 22, 2023
7
Definitions of Rationality ….2• Moreover, if consciousness is not stipulated as an
element of rationality, are only deliberate processes of adaption admitted, or non-deliberate ones as well? The typist trains herself to strike a particular key in response to the stimulus of a particular letter. Once learned, the act is unconscious, but deliberate. On the other hand, any person instinctively withdraws a finger that has been burned. This is “rational” in the sense that it serves a useful purpose, but is certainly neither a conscious nor deliberate adaption.
April 22, 2023
8
Definitions of Rationality ….3• Shall we moreover, call a behaviour “rational”
when it is in error, but only because the information on which it is based is faulty? When a subjective test is applied, it is rational for an individual to take medicine for a disease if he believes the medicine will cure the disease. When an objective test is applied, the behaviour is rational only if the medicine is in fact efficacious.
• Simon, H. A, (1997) Administrative Behaviours: A Study of Decision-Making Processes in Administrative Organizations. New York: The Free Press.
April 22, 2023
9
Clarity of argument http://www.online-utility.org/english/readability_test_and_improve.jsp
Fog Index 25.2 (p63)
22 April 2023Getting Academic Papers Published
10
22 April 2023Getting Academic Papers Published
11
April 22, 2023
12
Two major approaches to rigor•Rigor may be seen as a function of
process This is the tick the box approach where we
are checking that all the necessary tasks have been appropriately addressed
•Rigor may be seen as a product of evaluation
To be able to apply rigor as an evaluation you first have to performed a rigor checklist type approach
Then you need to have a standard to which to compare the research under review
April 22, 2023
13
Rigorous websites• http://www.wjh.harvard.edu/nsfqual/Ryan%20Paper.p
df --- What Are Standards Of Rigor For Qualitative Research? Gery W. Ryan RAND Corporation
• http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf--- Understanding Reliability and Validity in Qualitative Research , Nahid Golafshani, University of Toronto, Toronto, Ontario, Canada
• http://www.nova.edu/ssss/QR/QR7-4/pare.html---Enhancing the Rigor of Qualitative Research: Application of a Case Methodology to Build Theories of IT Implementation, Guy Paré+
• http://faculty.babson.edu/krollag/org_site/craft_articles/staw_rel_rigor.html --- Staw, B. M. (1985) Reports on the road to relevance and rigor: Some unexplored issues in publishing organizational research. In L. L. Cummings & P. J. Frost, eds. Publishing in the Organizational Sciences (pp 96-107). Homewood Illinois, Richard D. Irwin
April 22, 2023
14
Some more useful rigorous websites•http://www.youtube.com/watch?v=SW9N3
4LH7KQ a different use of the word rigour.
•http://www.youtube.com/watch?v=RZ4a8pQIwfE a general remarks
•http://www.youtube.com/watch?v=Nnp4qZX9H1U in mathematics
April 22, 2023
15
Some questions about the context of rigour•Has an adequate case been made for the
conceptual framework of the research?•Is there a convincing case for the
importance and relevance of the research question?
•Does the data required as specified by the researcher resonate with the research qustion
•Has a well established methodology/method been chosen or has a methodology/method been created by the researcher?
April 22, 2023
16
Characteristics of a rigor•Careful and precise uses of definitions • Making any assumptions used visible•Avoiding any obfuscation •Step by step logical path•Exposition demonstrated by examples•Heightened sense of being critical•Has constant comparison been a
characteristic of the study
April 22, 2023
17
Rigor as a process----1• What are the processes of academic research?
Producing a suitable research question Demonstrating the position of the research question within the extant literature
Establishing the connection to theory or arguing for grounded theory
Identifying an appropriate research methodology and discuss its advantages over other possibilities
Collecting/gathering/developing data/evidence
Detectives are Researchers
April 22, 2023
19
Rigor as a process----2 Managing the data and the software required (if any)
Analysing the data and producing the results
Postulating our finding and conclusions
Discussing the research through reflection including using the audit train and arguing for its relevance and rigor
Developing management and/or business guidelines
April 22, 2023
20
Producing a suitable research question•We are not concerned here whether the
research question is interesting, topical or useful.
•We are concerned to see if the research question is clear, unambiguous and answerable and that it is associated with some theory.
•The same thinking will also apply to sub-questions.
April 22, 2023
21
The purpose of literature in research•A knowledge of the literature is of
paramount importance to academic research for 2 key reasons.
The literature positions the research in terms of what is known in the field of study about the topic and way of researching in that field.
The literature is used to call on authority to support the arguments made by the researcher. Be careful of the authority you invoke as some of the material published is wrong.
April 22, 2023
22
Demonstrating the position of the research question within the extant literature• All academic research is underpinned by the
extant literature.• The key rigor issues here are:-
Peer reviewed journals need to dominate the reference list.
A significant number of the papers need to be drawn from highly rated journals (ABS) http://www.the-abs.org.uk/?id=257
The majority of the papers should be contemporary Do not rely too heavily on an author or one journal Other sources such as books and less formal
publications may be use sparingly
April 22, 2023
23
Identifying an appropriate research methodology and discuss its advantages over other possibilities
1. Research methodology needs to be clearly articulated and rooted in the researcher philosophical view point
2. The chosen research methodology needs to be justified on its own and
3. It has to be justified relatively to other methodological options.
April 22, 2023
24
Collecting/gathering/developing data/evidence
•For the purposes of your research data needs to be defined
•A philosophical stance on the nature of data collection is important
•At least two philosophical understandings of data need to be considered:-
Data of a product of mining – this has nothing to do with data mining.
Data as a product of travelling
April 22, 2023
25
Managing the data and the software required if any
•There are issues related to the storing and management of the data and keeping hard and soft copies
•Data will normally be coded and transferred to a computer
•Version control is imperative as is backup copies
•Software selection is important and the need for skills required in using the software is important
April 22, 2023
26
Analysing the data and producing the results
• Rigor relies on a demonstration of being able to use appropriate data analysis to explore the research question/s▫What was the nature of the data used?▫How much data was used?▫How were data suppliers chosen?▫From whom was it collected and how many?▫What was the stance of the data suppliers?▫Have the assumptions underpinning the analysis
been complied with?▫Are the results significant and in what way?
April 22, 2023
27
Postulating our finding and conclusions
•The findings of the data analysis need to be clearly articulated
•The findings need to be interpreted•The researcher needs to point out what
the findings suggest and what they do not suggest
•This requires skill at rhetoric
April 22, 2023
28
Developing management and/or business guidelines
•Research findings are not the same as the results of research
•One of the problems which sometimes occur in the presenting of business guidelines is the apparent attractiveness of motherhoods. Motherhoods have to be avoided. They are superficial generalities which have no place in academic research.
April 22, 2023
29
Experimental and Quasi-experimental•The so-called gold standard of rigor in
experimental research is to use a random sample and to have a control group.
•A quasi-experiment does not fully comply with the rigor required for an experiment. The difference is that in a quasi-experiment the participants/informants will not have been randomly chosen.
April 22, 2023
30
Rigor as a product of evaluation•How do we do this?•Any evaluation means comparing what is
being evaluated to a standard---- Here we are looking for a gap between what we have and an ideal of what we want i.e. standard----We need the “application of precise and exacting standards”
•We need to avoid shallow analysis•What issues in academic research need to
be evaluated?
April 22, 2023
31
Issues in academic research to be evaluated
•Each chapter has a list of issues to be examined
•Each issue needs to be checked for completeness and for depth and where possible the conclusions need to be supported by authority
April 22, 2023
32
Part 2
April 22, 2023
33
Analytical rigor•The need to avoid shallow analysis •How do we tell what is rigorous•Rigor is following the rule -- too simple to
be useful
•How do you assess the quality of an analysis? Rigor starts with getting the research question right. The research question leads to the hypothoses or propositions.
April 22, 2023
34
Some issues• The breath and depth of exploration of alternative
analysis.• How much critiquing has been done.• 8 different kinds of characteristics of a Meta
judgement of the rigor of an analysis The facts are nothing more than ………reading the news Reconsidered from diverse points of view Build a chain of reasoning How has the reasoning and logic been checked Quickly generating a hypothesis is an indication of low rigor Avoid closing the issue too soon. Pre-mature closure Are you going to have the best hypothesis on the topic. Find on topic material ---broadening check
April 22, 2023
35
A multi-attribute measure of sufficiency
•Rigor is a product of the culture of the school
•A revised definition of rigor, reframing it as an emergent multi-attribute measure of sufficiency rather than as a measure of process deviation.
April 22, 2023
36
What is meant by shallow analysis?•Shallow analysis is not adequate to
deliver a fuller meaning of the work which has been performed.
•Shallow analysis is incomplete.•Shallow analysis may miss the point of the
whole research or findings.
April 22, 2023
37
What is meant by shallow analysis?..11. A poor set of initial papers.
The most appropriate papers were not read and considered.
2. An inadequate critique of the literature?
3. There is an unsatisfactory research question/s and this hypothesis
4. The importance of the research was not adequately argued
5. The methodology was not adequately argued
1. There are clear rules about this issue
2. More subjective 3. The test here is to seek
support from supervisor, colleagues and outside influencers
4. The force of the argument is the main issue here
5. There are clear rules about this issue
April 22, 2023
38
Hypothesis -- the world is getting warmer – Global warming?
April 22, 2023
39
Support• Make a list of all those who support your research
besides yourself and your supervisor.• Have you encountered any paradoxes or
contradictions in your research so far?• Have you any lurking doubts about the way your
research has gone so far? • Lurking doubts are a central issue in academic
research. If you have no lurking doubts you do not understand the problems/challenges you are facing Remember the aphorism “if you are not confused and fearful you do not understand the issues you are facing”.
April 22, 2023
40
What is meant by shallow analysis?...21. An unsatisfactory sample was
used – no cherry picking2. Are the analytical procedures
been conducted properly conducted? Any biases?
3. Have the results of the analysis being understood correctly? What alternative candidate explanations have been considered?
4. Is there a convincing discussion? Has there been any supporting information.
5. Have the results been convincingly converted to management guidelines or policy options?
1. There are clear rules about this issue
2. There are clear rules about this issue
3. This is a more creative issue
4. This is a function of rhetoric and support from supervisor, colleagues etc
5. This is a creative issue and there is no cookbook answer
April 22, 2023
41
What is meant by shallow analysis?....31. Has an adequate
suggestion been made concerning future research?
2. Has adequate attention been given to the principle of parsimony?
3. Is the written language at an acceptable standard?
4. Have all the universities requirements been complied with regard presentation?
1. This is a more creative issue
2. There are clear rules about this issue
3. There are clear rules about this issue
4. There are clear rules about this issue
April 22, 2023
42
Exploring challenges•The researcher needs to have thought
about a list of potential criticisms which examiners/reviews may use
April 22, 2023
43
Introduction• Is the topic of the
research clear ?• Is the objective of the
research clear ?• Are all the terms
adequately defined?• Has a possible theoretical
context been highlighted?• Has the importance of the
research been argued?• Has the course of the
research been described?
April 22, 2023
44
Literature• Have all the important
authors in the field been included?
• Is the literature being explored on a concept or timeline basis or a combination of both and why has the approach been chosen?
• Is there an adequately critical approach to the literature?
• What gaps are there and how does the research question relate to the gaps?
April 22, 2023
45
The methodology was not adequately discussed and argued• What type of investigations
is being proposed and why?• What is the unit of analysis?• What time horizon is being
suggested?• What data/evidence is
required?• What type of instrument is
needed?• How will the data be
analysed?• How will bias be minimised?• Alternatives?
April 22, 2023
46
The research• What did the research
protocol look like?• How were the ethical
issued addressed?• What was actually done?• How were informants
chosen?• How may?• Where?• How was the
data/evidence collected?• What data management
issued arose?
April 22, 2023
47
Findings and conclusions• Was the analysis
conducted with appropriate knowledge and care?
• What do the findings mean?
• What are the limitations of the analysis?
• How were the conclusions drawn from the research?
April 22, 2023
48
Audit trail and reflections• How did the research
process develop?• What were the major
miles stones in the thinking?
• How has the researcher developed over the period of the research?
• Looking over the whole experience how does the research feel about what has been achieved?
April 22, 2023
49
Management guidelines and future research• So what?• What’s new?• Who cares?• How can the research
finding be put to use?• What are the other
interesting issues encountered during the research process?
Zen and the Archer• Many years’ ago in Japan, there was a warrior – one of
those itinerant Samurai known as ronin. He had an ambition to find fame and fortune as the finest archer in the land. In pursuit of his dream, he travelled the length and breadth of the country looking for a master-bowman to help him improve his technique. Few of those he encountered had much to teach him but he continued searching and eventually, just as he was about to give up, he chanced upon some dilapidated farm buildings far from other habitation. All over the buildings, in the most unlikely and inaccessible places, were hundreds of hand-painted targets, each with an arrow at the exact centre of the bulls-eye.
Zen and the Archer• The Samurai knocked at the door and begged the
occupant to share the secret of his uncanny accuracy. A bargain was struck and the master-bowman offered to demonstrate his craft. Pausing a few moments to slow his breath and meditate, he nocked an arrow onto the string and – in the classic stance – drew the well-worn bow. With the sudden gentleness of a young child releasing its hold on an adult’s finger, he loosed the arrow towards the barn where it came to rest, planting itself into… a perfectly blank stretch of the wall.
Zen and the Archer• “You missed,” said the Samurai.• “Not so,” replied the bowman, stepping up to the
barn, picking up a paintbrush and marking a target round the quivering arrow. “Every time, a bulls-eye.”
• “But why do you do this?” asked the stunned warrior.• “Because I am intensely curious to learn where every
arrow lands,” said the bowman.• In that moment, the Samurai was enlightened. He
forgot his dream of fame and fortune and became the master-bowman’s pupil in the art of Zen archery.
