Peer Feedback and Methodological
Improvement in Lablet Research: A
Qualitative Analysis
Lindsey McGowen, PhD
Lablet Evaluator
Christine Burgh
Research Assistant
North Carolina State University
June 2, 2016
Lablet Goals
• Solve hard problems in cyber security
• Develop and conduct methodologically rigorous research
• Develop a community of practice for SoS
Methodology Support Activities
• IRN-SoS
• SoS methodology benchmarking study
• Methodology consulting
• Methodology guidelines
• Methodology feedback seminars
Methodology Feedback Seminars
• Format:
– Students present their research
– Lablet researchers provide feedback
– Feedback is recorded via an online
– Presenters receive a feedback report
Methodology Feedback Seminars
• Seminar format piloted in Spring 2014
– Adjustments made each semester based on
participant feedback
• Seminars held Fall 2014 – Spring 2016
– 25 seminars
– 31 research presentations
– 33 presenters
Methodology Feedback Seminar Impact
• Presenters report
they used to improve
their research
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Feedback Used
Yes No
Methodology Feedback Seminar Impact
• Presenters report
changes were made
primarily to the study
Intro & Background
and Methodology
0%
20%
40%
60%
80%
100%
Changes Made
Intro & Background Methodology
Analysis & results Conclusions
Language & Style
Methodology Feedback as Data
• A rising tide raises all boats?
• Feedback reports used as evidence to investigate
whether lablet research is becoming more
methodologically rigorous over time
• Why qualitative coding
– Captures the perspectives if Lablet researchers
– Experts in cyber security research
– Enhanced validity of conclusions about the
methodological rigor of research
Research Questions
• RQ1: On which aspects of the research process does
Lablet methodology feedback reflect a need for
improvement?
• RQ2: Does the feedback provided in Lablet methodology
seminars change over time?
• H1: Feedback from early semesters will focus on more
fundamental aspects of the research process.
• H2: Feedback from later semesters will focus on deeper
and more nuanced aspects of the research process.
Qualitative Methodology
• Feedback reports entered into a database
• Data fields based on methodology guideline
components:
– Abstract/Overview
– Background/Lit Review
– Research Goal
– Methodology
– Analysis/Results
– Conclusions
– Language/Style
– Feedback from
respondents N =
102
Qualitative Methodology
• Upload Data to qualitative coding software: Dedoose
Qualitative Methodology
• Qualitative Coding:
– Process of identifying meaning within the data and
creating descriptive labels or “codes” that can be
applied to the text
– Codes grouped into topics and themes
– Allows voluminous text to be segmented and
categorized into manageable units
– Allows for comparison and analysis across diverse
research projects
– Codes based on methodology guidelines and content
analysis
Qualitative Methodology
• Code Development
Process
– Select 50% of the data
– 2 coders
independently identify
themes within the data
and develop codes
– Compare codes
– Combine similar codes
– Reconcile differences
to arrive at a common
set of codes
– Develop code
definitions
– Iterative process• 4 iterations so far…
• Coding for Methodology
entries complete
• Coding for other aspects
of the research on-going
– Apply codes to 100%
of data
– Calculate inter-rater
reliability
Inter-Rater Reliability
• Cohen’s kappa statistic—Cohen (1960), ‘A coefficient of
agreement for nominal scales.’
– Widely used and respected measure to evaluate
inter-rater agreement as compared to the rate of
agreement expected by chance—based on the
coding behavior or each rater.
– Values above .80 are considered to be excellent
agreement (Cicchetti,1994; Fleiss, 1971; Landis and Koch,1977; Miles and Huberman, 1994)
• Cohen’s kappa for “Methodology” entries = .91
Codebook
• Coded “Methodology” entries, but responses spanned
wide range of research process aspects
• 3 themes
– Positive Codes: good, clear
– Improve Codes: specify, clarify, justify, improve
– Specific suggestions
• 12 research topics
• 53 sub-topics
• 206 unique codes
Codebook
Coding Examples: Background
• Ex: Clarify contribution to the field
– “try to characterize what new information you are
trying to achieve. High level characteristics of
vulnerabilities in [aspect of a cyber system]. You can
then create an artifact to automatically triage [aspect
of cyber system]. Can expand the same ideas to VCL,
EC2, etc.”
– “How is your framework different from these prior
approaches and how does it compare”
Coding Examples: Threat Model
• Ex: Specify threat model
– “The authors should determine what the threat model
is and determine adversarial models. This is an
important step in evaluating the effectiveness of the
proposed approach. Note that it is reasonable to not
evaluate an adversarial model at first, but it eventually
needs to be studied.”
• Ex: Clarify threat model
– “Improved clarity of the threats models faced in this
approach would help.”
Coding Examples: Research Goal
• Ex: Clarify research goal
– “You talked about the research goal, not clear if
preventing malicious automation or uniquely ID a
user”
• Ex: Clear research questions
– “Good to explicitly state research question”
– “Good identification of research question”
Coding Examples
IRB
• Ex: Clear IRB
– “Good to mention IRB”
Methodology
• Ex: Clarify metrics
– “Needs clear statement of metrics”
– “question about quality metric – is there any
subjectivity? Question about effort metric – concern
about using likert scale to multiply”
Coding Examples: Methodology
• Ex: Suggest within-subjects design
– “Use everyone as their own reference, and control for
individual differences that way. Have people do at
least 2 scenarios”
• Ex: Justify data source
– “the presentation could have better motivated the
value of using [app development site] data”
– “some more discussion may be needed to talk about
potential biases introduced by sampling through
Mturk”
Coding Examples: Methodology
• Ex: provide logical justification
– “Use first order logic”
• Ex: Clarify experimental condition
– “What is the experiment environment? Needs to be
described in detail”
– “This is a variant of the prisoners dilemma challenge,
can you talk more about that?”
– “Are you using these 2 scenarios for your study?”
Coding Examples: Analysis
• Ex: Clarify analysis steps
– “Clarify the steps you take in your analysis.”
• Ex: Specify analysis steps
– What tests are you running?
• Ex: Good success criteria
– “Nice game setup and methodology, success criteria
and metrics”
• Ex: Add statistical control variables
– “learnability is confounded with reading speed - either
drop slide time or measure reading speed for each
participant and statistically control for that. “
Coding Examples: Conclusions
• Ex: Specify next steps
– “I understand how the completed experiments were
performed, but I don't think we had enough time to
discuss future work. I'm curious how future work will
bring the experiments closer to practical user
authentication.”
– “Ideas for future work - There was a great discussion
of how the overlay filesystem used by [system
platform] may be able to be leveraged to
automatically push security updates to children
images.”
Coding Examples: Resource
• Ex: Suggested resource
– “Is anyone aware if anyone has studied to see if
people are desensitized to security threats b/c of it's
news coverage. Dr. Statton was doing a sentiment
analysis, that's the closest I've heard.”
– “I'd definitely consider citing or at least considering
some of the work exploring the limitations of MTurk
participants. Here's links to a couple...”
Coding Examples: Validity
• Ex: improve external validity
– “Game shows the probability of the next threat type,
but in the real world you wouldn’t have that
information”
– “best practices are only a starting place. If you talk to
someone running the security operations for an
enterprise, they will tell you that the best practices are
necessary, but not sufficient They will want to go
much beyond best practices and customize a solution
to their environment”
Coding Examples: Validity
• Ex: Specify threats to validity
– “what are the treats to validity of your research? must
be some threats, due to unreported vulnerabilities, or
increase in programmer skill that could account for
findings. any alternative explanations for what you
observed?”
– “Once additional piece of context, you always did this
at the end of the semester, sometime in the middle of
the semester we do a whole section of security
requirements… I wonder if [we] are somehow
confounding your results by teaching the class…”
Coding Examples: Language & Style
• Ex: Good graphics
– “nice graph to describe methodology/process of data
collection and measurements”
• Ex: Use precise language
– “use similar/same language so that reader is not
confused.”
– “when describing which systems you will look at, refer
to a specific sampling procedure. if you have
selection criteria (such as availability of data, etc)
refer to them as selection criteria.”
So is there any evidence from the feedback that Lablet
research is becoming more methodologically rigorous over
time?
Preliminary Results: Themes
Fall2014 Spring2015 Fall2015 Spring2016 Total
Positive 13% 13% 9% 15% 14%
Improve 78% 77% 83% 74% 77%
Suggestion 9% 10% 8% 11% 10%
TOTAL 100% 100% 100% 100% 100%
No clear patterns in the data so far…
Preliminary Results: Research Topic
Emerging patterns in the data?
Fall2014 Spring2015 Fall2015 Spring2016 TotalBackground 8.7% 7.1% 14.5% 4.7% 7.4%Threat Model 0.0% 0.0% 1.8% 0.6% 0.4%Research Goal 13.0% 13.1% 14.5% 10.6% 12.3%IRB 0.0% 2.4% 0.0% 1.2% 1.2%Methodology 52.0% 42.8% 34.5% 41.8% 43.1%Analysis 13.0% 7.1% 7.3% 8.8% 8.8%Conclusions 0.0% 1.2% 1.8% 1.2% 1.0%Resource 0.0% 2.4% 1.8% 3.5% 2.3%Validity 8.7% 13.1% 20.0% 22.9% 16.7%Language and Style 4.3% 7.1% 1.8% 4.1% 4.9%Study Description 0.0% 2.4% 3.6% 1.2% 1.7%Unintelligable due to Typo 0.0% 1.2% 0.0% 0.0% 0.4%Total 100.0% 100.0% 100.0% 100.0% 100.0%
Some Observations
• Still need to code entries for several more research
aspects
– Abstract/Overview, Background/Lit Review, Research Goal,
Analysis/Results, Conclusions, Language/Style
• Data entry fields do not appear to correspond to the type
of feedback entered in those fields
• Data skewed toward Spring 2016: 45% of all entries,
36% of all codes
– Much better attendance: Seminar became a course
and faculty scheduled their students
Next steps
• Migrate to NVivo software
– Dedoose is glithcy
• Code additional data entry fields
• Further refine codebook
– Would like to have CS coder(s)
• Analyze data for changes in feedback over time
QUESTIONS?