The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments. Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words. • In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents.
7
Open Surveys and their analysis David Alman October 2012 Version 3 Page 1 Open Surveys and their analysis 1. Open Surveys The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments. Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words. Open Surveys are very different to what are termed Statistical Surveys for, in a sense, they work in reverse. That is to say: In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents. In comparison; Statistical Surveys use a “forced” response approach to questions that are structured around a pre-agreed model. These models have factors (categories) that are used as the basis to develop questions around an area of interest. Respondents have little to no opportunity to explain their answers. The answers provide a means of statistically (quantitatively) confirming responses against a survey’s underlying model’s factors (categories). As Statistical Survey’s generally do not explore questions in detail or depth, this can affect how valid they are where issues are more complex and need respondents to explain reasons for believing something, as is available through Open Surveys. Open Surveys are also termed Open Question Surveys and use respondent comments that are sometimes described as Narrative data; Text data; or Qualitative data. 2. Advantages and disadvantages of Open Surveys Advantages: Allows respondents to answer in their own words. Allows for "richness" or "depth" of explanation in responses. Can identify and explore issues not currently fully recognised or understood. Can identify options for further action. Disadvantages: Limited by the writing skills of respondents. Analysis involves a more complex interpretation. Stronger role of interpretation in analysing responses.
Transcript
Open Surveys and their analysis
David Alman October 2012 Version 3 Page 1
Open Surveys and their analysis
1. Open Surveys The purpose of Open Surveys is to help understand and improve the effectiveness of an organisational change or some aspect of organisational performance based on respondent comments. Open Surveys and their analysis are based on, and developed from, respondents’ thoughts and feelings expressed in their own words. Open Surveys are very different to what are termed Statistical Surveys for, in a sense, they work in reverse. That is to say:
In an Open Survey respondents answer a few questions in detail by expressing themselves in their own words. Responses are grouped into categories, and from these categories broader patterns are built showing how comments are linked. Such patterns are, or can be fitted together, into a model covering and representing the collective views of respondents. In comparison;
Statistical Surveys use a “forced” response approach to questions that are structured around a pre-agreed model. These models have factors (categories) that are used as the basis to develop questions around an area of interest. Respondents have little to no opportunity to explain their answers. The answers provide a means of statistically (quantitatively) confirming responses against a survey’s underlying model’s factors (categories).
As Statistical Survey’s generally do not explore questions in detail or depth, this can affect how valid they are where issues are more complex and need respondents to explain reasons for believing something, as is available through Open Surveys. Open Surveys are also termed Open Question Surveys and use respondent comments that are sometimes described as Narrative data; Text data; or Qualitative data. 2. Advantages and disadvantages of Open Surveys Advantages: Allows respondents to answer in their own words. Allows for "richness" or "depth" of explanation in responses. Can identify and explore issues not currently fully recognised or understood. Can identify options for further action. Disadvantages: Limited by the writing skills of respondents. Analysis involves a more complex interpretation. Stronger role of interpretation in analysing responses.
Open Surveys and their analysis
David Alman October 2012 Version 3 Page 2
3. Designing Open Survey Questions Questions used in Open Surveys usually use a word such as how, what, when, where, and why, to allow respondents to express their thoughts and feelings such as their attitudes, opinions, understanding, likes, dislikes, suggestions, and ideas. Open (sometimes referred to as unstructured) questions are ones in which possible answers are not suggested in the design of the survey questions, and where respondents answer in the way they see the world. Open ended questions are framed to encourage self explanation of thoughts and feelings in a sentence, paragraph, or more. Open ended questions include some direction on which the feedback is sought, such as:
Please tell us what changes you would like to see?
Please tell us what in your view we do well?
Where could we improve?
Why do you use our services? 4. Analysing Open Survey Responses 4.1 Open Survey and Analysis Framework The purpose of an Open Survey design and analysis of responses is to provide a framework that: a) Accurately and as thoroughly as possible captures the intent of respondent comments;
and b) Analyses these comments so that they construct, step by step, an understanding of
issues raised in a way that is both meaningful and relevant: That “fit and work” [1]. An explanation of how respondent comments are analysed, developed, and tested is provided in 4.2 Steps in Analysing Comments. 4.2 Steps in Analysing Comments Step 1. Read through all comments to get a feeling for the responses, and themes that emerge from this overview. Step 2 Develop Categories 2.1 Create categories. Identify categories (sometimes referred to as labels) from the different themes that emerge from comments. When a response is allocated to a category this is based on what is actually said and also any underlying meaning that might be recognised. 2.2 Test to improve Category Analysis. Test to see if there are alternative categories that have a better fit.
Open Surveys and their analysis
David Alman October 2012 Version 3 Page 3
2.3 Decide if there are sub categories. Think about what the categories are about. Once comments have been categorised look again at the responses in each category to identify what is being expressed by different viewpoints. Are these consistent with the category, and could they also identify sub categories that are of value in the analysis? 2.4 Quantify Categories. Assign comments to at least one category (or sub category) and group them so the quantity of responses in each category can be more easily counted. Refer to Diagram 1. Developing Categories from Respondent Comments. Diagram 1. Developing Categories from Respondent Comments Step 3 Link Categories to develop Patterns and Model(s) Once comments have been studied and categorised (and sub categorised), the next step is to see how these categories link to form patterns and models:
Do some categories link in some way, and how do unrelated others link to form patterns?
Do these patterns, together, form to represent one or more models? In developing patterns check if there are exceptions to the rule that require the patterns to be broadened to include other categories, or changed into other patterns, or cast doubt over a pattern as a “rule”. Refer to Diagram 2. Developing Patterns from Categories.
Respondent Comments
ID All Survey Comments 54 My supervisor is not
supportive and criticises me in front of others.
84 There is a lot of rework causing delays.
165 Teams and Divisions are unwilling to work together.
248 Work processes are inefficient and bureaucratic.
362 There are conflicting work objectives and targets.
420 Work output is more important than people.
Total 432
Category: Role performance
ID Comments 84 There is a lot of rework causing delays
Total 220
Category: Organisational objectives
ID Comments 362 There are conflicting objectives and
targets
Total 63
Category: Work Relations
ID Comments 165 Teams and Divisions are
unwilling to work together
Total 116
Category: Work processes
ID Comments 248 Work processes are
inefficient and bureaucratic
Total 260
Open Surveys and their analysis
David Alman October 2012 Version 3 Page 4
Diagram 2. Developing Patterns from Categories In developing models these may be either:
a) Drawn from the patterns of analysis; b) Brought to the analysis from a recognised published model that links to categories
and their patterns. For example a recognised model relevant to the type of survey carried out such as a Hospitality, Health Service, Service Delivery, Management, Environmental Management, or Community Model.
Either approach to developing a model offers advantages, such as an internally developed model can assist in gaining commitment from those involved in its development whereas the use of an external model offers validity through its credibility as a recognised benchmark. Whichever approach is used the selected model should be as inclusive as possible of all respondent comments. Refer to Diagram 3. Organisational Alignment Model based on fit to Patterns.
Pattern of Performance Issues Pattern of Cultural Issues
Category: Organisational objectives
Category: Work Processes
Category: Job Performance
Category: Job relationships
Category: Work Relations
Category: Organisational values
Open Surveys and their analysis
David Alman October 2012 Version 3 Page 5
Diagram 3. Organisational Alignment Model based on fit to Patterns Step 4. Write up the analysis: Once all comments are categorised and analysed, and the patterns they form identified along with any model that fits those patterns, then the next step is to write up in a summary. The analysis can include:
An explanation of a model and the patterns that pull the analysis together.
Categories and sub categories identified (at the very least the more significant categories).
The number of comments covered by a category. This can be in either broad quantitative terms covering more than one category, or specific to each category.
An explanation about the categories and sub categories along with supporting (non identifiable) quotations drawn from respondent comments.
Recommendations based on the analysis. Recommendations for improvements can come directly from comments where there are several similar responses or from a single, different, comment.
The survey design and analysis should demonstrate validity:
Face Validity: o How survey findings make sense in terms of credibility, relevance, and usefulness
to respondents and those who decide to use survey recommendations; o Use respondents own words in providing credibility to survey recommendations.
Organisational Values
Organisational Objectives
Work processes
Work relations
Job performance
Job relationships
Organisational Culture
Organisational Performance
Open Surveys and their analysis
David Alman October 2012 Version 3 Page 6
Construct Validity: The extent to which the survey design and analysis minimise error and misinterpretation and is consistent in “fitting” with respondent comments. That is, where comments fit with constructs (or concepts) in terms of developing criteria; patterns; and models.
Notation. [1] In references below, see Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey Responses, the term Concept Mapping is used. The term Construct Mapping is used here instead of the term Concept Mapping so that there can be a better and more logical alignment to Construct Validity. In this respect an Open Survey analysis could be viewed as a form of Construct Mapping in that comments are organised (i.e. constructed) first into categories; these categories are then organised (constructed) into patterns; and where these patterns may then be organised (constructed) into a model. When constructing such a map the outcomes (whether this relates to the construction of categories, patterns, or a model) should readily, not forcibly, “fit” with respondent comments and also “work” in the sense that categories, patterns, and any model selected can be seen to provide meaningful and relevant conclusions. “Fit” in this sense relates to Construct Validity, and “work” to Face Validity. References include: Outline explanations of Open Survey Analysis Frameworks Analysing open-ended questions. Website: http://intelligentmeasurement.wordpress.com/2007/12/18/analyzing-open-ended-questions/ Downloaded 8/8/12 Analysing Qualitative Data. Website: http://learningstore.uwex.edu/assets/pdfs/g3658-12.pdf Downloaded 8/8/12 A Brief Guide to the Analysis of Open-Ended Survey Questions. Website: http://cms.cerritos.edu/uploads/ResearchandPlanning/Brief_Guide_to_Open-Ended_Survey_Questions.pdf Downloaded 8/8/12 Concept Mapping Explained Concept Mapping as an alternative Approach for the Analysis of Open-ended Survey Responses Website: http://www.socialresearchmethods.net/research/Concept%20Mapping%20as%20an%20Alternative%20Approach%20for%20the%20Analysis%20of%20Open-Ended%20Survey%20Responses.pdf Downloaded 8/8/12 Comprehensive References on Open Survey Design and Analysis Qualitative Data Analysis. Ian Dey. Website: http://www.drapuig.info/files/Qualitative_data_analysis.pdf Downloaded 8/8/12 Patton, M.Q. (1990). Qualitative evaluation and research methods (2nd ed). Newbury Park, California: SAGE Publications Inc. Organisational Alignment Model in Diagram 3 adapted from Figure 3.9 in:
Addison, R., Haig. C., & Kearney. L. (2009). Performance architecture: The art and science of improving organizations. San Francisco, California: Pfeiffer. Key words Open surveys; Open survey analysis; Construct Mapping; Concept Mapping
