Research methods, EngD course, 2012 Slide 1

Research methods for socio-technical analysis

Ian Sommerville

Research methods, EngD course, 2012 Slide 2

Human and organisational issues

• When designing, testing or evaluating critical systems we need to know about how that system will be used in real settings

– What do users really do and why?– What potential is there for error?– How are errors avoided or mitigated?

• There are many useful generic insights into human factors – For example the GEMS model

• But systems engineers will inevitably need to understand human factors that are specific to the system they are constructing or maintaining

– Specifics emerge from: the location, the people, their work, the technologies in use, etc.

• Process models are too abstract to tell us what we need to know about what users do

Research methods, EngD course, 2012 Slide 3

Human and organisational issues

• Because of the importance of human and organisational factors, systems engineering draws from fields such as

– psychology – sociology – organisational studies

• But draws from their methods as much as their results– Don’t expect research done in other disciplines to provide all the

necessary answers for questions in systems engineering – Don’t expect that researchers from other disciplines will understand

or be interested in technology systems. Some are, many aren’t

• Systems engineers have to– Find out about human and organisational factors themselves– Work closely with people from other disciplines

Research methods, EngD course, 2012 Slide 4

The devil is in the detail• The fundamental principle of software design

is abstraction– Find the commonalities between the entities in the

system and represent these as software elements• This is based on the assumption that the

common features are the most important and that stakeholders can adapt to them

• Social science perspective is rather different. The focus is on the detail of a situation

– The key argument is that the details are very significant indeed and that failure to pay attention to them leads to problems

Research methods, EngD course, 2012 Slide 5

Studying socio-technical systems

• Systems engineers need to find out about about human, social and organisational issue that are specific to the system they are working on and to the class of systems being developed

• There are several useful ways to do this1. Experiments2. Archive research / Case studies3. Interviews and focus groups4. Simulation5. Fieldwork

• And several less useful ways (which are better than nothing)1. Questionnaires and surveys2. Imagining what users do

Research methods, EngD course, 2012 Slide 6

Case study analysis• Retrospective analysis, looking back at a

system or incident (a case) that has occurred• In-depth study, based on a detailed analysis of

a set of information sources• Based around one or more research questions

e.g.– What were the factors in system X that led to

catastrophic failure?– What management decisions contributed to the cost

overrun in system Y?• Intention is to gain generally useful insights

from specific cases

Research methods, EngD course, 2012 Slide 7

Examples of case studies• “Normal accidents’

– The problem of nuclear reactor overheating at 3-mile Island

• Denver airport baggage system• The London Ambulance Computer-aided

Dispatch System• Airbus accident – Warsaw airport• Case studies tend to focus on system failure

rather than system successes

Research methods, EngD course, 2012 Slide 8

Stages of case study analysis

• Determine and define the research questions• Select the cases and determine data gathering

and analysis techniques• Prepare and collect the data• Collect data in the ‘field’ (if appropriate to do

so)• Evaluate and analyse the data• Prepare the report of the case study

Research methods, EngD course, 2012 Slide 9

Research questions• Establish the focus of the study by forming questions

about the situation or the problem to be studied and so defining the study’s purpose

• The question or questions are then used to select material for analysis

• Accident enquiries are focused case studies where the question is decided in advance i.e.

– Why did accident X occur?• Danger of bias in establishing the research question

– What management problems led to the collape of Lehmans?

– What were the systemic factors that led to the collapse of Lehmans?

Research methods, EngD course, 2012 Slide 10

Data gathering and analysis techniques

• Case study selection– Often the most ‘typical’ case study does not reveal the

greatest insights– Choose case studies that are

• Extreme - representative of an exceptional situation e.g. an accident

• Critical - having strategic importance with respect to the general problem

• Data to be analysed– Essential to use multiple sources of data

• Interviews

• Archives and reports

• Fieldwork

Research methods, EngD course, 2012 Slide 11

Data collection and analysis• Typically, case studies generate large amounts of

qualitative data– Transcripts of interviews– Reports and associated notes

• Look for opportunities for triangulation– Evidence from diverse sources that supports the same

conclusions• Field data is also important

– Look at the object of the case study as it is– The system in operation

• Analyse the data in different ways and from different perspectives

– Use quantitative data to corroborate qualitative data

Research methods, EngD course, 2012 Slide 12

Case study issues• Generalisability

– Much of our learning is through case studies so we can make generalisations from them. However, we have to be aware of the possibility that specific factors may be dominant in any one study and be careful about generalisation

– Case studies can be useful for finding ‘black swans’ i.e. contradicting received wisdom

• Hindsight bias– Seeing events as more significant than they actually

were because there is a clear link between these events and the research questions

• Obsolescence of results

Research methods, EngD course, 2012 Slide 13

Fieldwork - Get Out of the Office!

Research methods, EngD course, 2012 Slide 14

Fieldwork• Generic name for a set of techniques which

involve collecting data directly from a worksite (the “field”), rather than relying on reports from other people

• Encompasses a range of techniques, which have much in common:

– Direct observation and discussion– Ethnography– Cultural and technology probes

Research methods, EngD course, 2012 Slide 15

Ethnography of work• Situated observation of work with a view to

developing a detailed understanding of the way that work is done and how the actors respond to other actors and the context of the work

• The aim is to develop a ‘rich picture’ of the work done

• Ethnomethodological ethnography– Ethnography that is intended to describe what work is

rather than explain work in terms of some social theory

– Approach proposed in the 1980s at Xerox, developed at Lancaster University in the 1990s.

– Work at Lancaster related ethnography explicitly to systems design.

Research methods, EngD course, 2012 Slide 16

Ethnography in practice• Looks at work in ‘real-time’ rather than

retrospectively• The ethnographer ‘lives in the workplace’ and

experiences it in the same way as the people involved in the work

• Examines the processes and artefacts that take place in the workplace

• The aim is to understand work as it really is rather than work as defined or retrospectively described

Research methods, EngD course, 2012 Slide 17

Examples of ethnography• The UK en route air traffic control system• The London Underground control room• Preparing and printing government documents• Health systems

– Managers in the health service– Breast cancer screening– NHS Direct

Research methods, EngD course, 2012 Slide 18

Fieldwork Methods• Get out of the office

– Talk to people about their work– Observe practices– Visit settings of use

• This can be part of an informal or formal study– How formal or informal you are depends on:

• The formality of the results you wish to produce (Are you writing a study to be read by others, or just trying to get a feel for a setting?)

• Your relationship to the people you research (Do they work in the same organisation as you? Are they customers?)

• The setting (Some workplaces require different kinds of formalities, e.g. steel mill workers vs. hospital workers)

Research methods, EngD course, 2012 Slide 19

Observe “ordinary practice”• Don’t wait for things to go wrong

– The routine and mundane are interesting (safe technology should be mundane and routine!)

– Error is a judgement, are you in a position to judge what is an error?

• What is supposed to happen and what really happens are often different

– People break the rules– People have to go to extra effort to make the rules work

• How is the system used in practice?

• What extra things are necessary to make the system work?

• Look and listen, but don’t judge. Users know their work better than you do

• Observation is sometimes referred to as “Ethnography”

Research methods, EngD course, 2012 Slide 20

Fieldwork and systems engineering

• “Traditional’ ethnography is based on the ethnographer spending prolonged periods in the workplace

– Problematical when there are deadlines for procurement and system specification

• Various techniques have been developed to address this issue

– Concurrent fieldwork– Evaluative fieldwork– Quick and dirty fieldwork

Research methods, EngD course, 2012 Slide 21

Concurrent fieldwork

Debriefing meetings

Ethnographicstudy

Systemsdevelopment

System prototype

Research methods, EngD course, 2012 Slide 22

Concurrent fieldwork• An approach that can be used in conjunction

with iterative development or system prototyping

– The field worker can act as a ‘surrogate user’ because of their knowledge of the work and workplace

– However, unlike users, they can be more objective about the system

• The debriefing meetings involve the fieldworkers and system developers

• The developers set questions for the fieldworker based on the prototype system

Research methods, EngD course, 2012 Slide 23

Evaluative fieldwork

Debriefing meetings

Ethnographicstudy

Amended specification or

design

Initial specification or

design

Research methods, EngD course, 2012 Slide 24

Evaluative fieldwork• An approach which is geared to ‘sanity

checking’ assumptions about a system or a system specification

• The field worker reads and understands the implications of the specification then makes observations of the processes to be suppprted

• During the debriefing meetings, he or she discusses the issues and problems that arose during the fieldwork and the team discusses possible changes to the specification to address these

Research methods, EngD course, 2012 Slide 25

Quick and dirty fieldwork

Debriefing meetings

Quick and dirty fieldwork

More detailed requirements

Outline requirements

Requirements engineering

Research methods, EngD course, 2012 Slide 26

Quick and dirty fieldwork• Short, focused studies to highlight the key

issues of concern in the ways in which work is done

• Aim of the work is to refine an initial specification

• Fieldwork is concentrated in a short period (1 day-1 week), with no iterations after debriefing meetings

Research methods, EngD course, 2012 Slide 27

Planning Fieldwork• Plan

– But don’t plan too much. Allow for the unexpected– How long you need is contingent on what you are looking at and

how long people will put up with you– Who you observe is also contingent on how long you have, and who

will have you. Don’t always assume the ‘important’ people are important (e.g. secretaries can often play critical roles)

– Whether you make recordings or take notes depends on the setting, and whether and how you want to write up examples afterwards

– Be ethical! This is not just about bureaucracy but about morality and common sense

• Access Issues– You may need to approach and negotiate with people: managers,

staff, trades-unions

Research methods, EngD course, 2012 Slide 28

Writing up results• What you do after a field study depends on who the results are

relevant to and how formal the work was– If it was sensitisation they may be nothing to do immediately, or

you may need to do a presentation or write a report and discuss it (Discussion is very helpful)

• The value of the sensitisation should be relevant later - it will influence subsequent decisions

– If the work was more formal and for wider dissemination (i.e. you work in a large team, a long project and/or your work has wide relevance) you will need to write more formal reports.

• First step should be to transcribe or write up what you have found

• Analysis can be done in a number of ways and is usually qualitative rather than quantitative

– Field-studies often lead to useful contextual information and often negative information rather than any functional or non-functional requirements.

Research methods, EngD course, 2012 Slide 29

• She works over constant periods – maybe a day, maybe 10 minutes.• Has done collaborations in past, but not now. Works for a ‘boss’.

Seems like the usual working arrangement. She has two supervisors, one bioinformatician, one biologist. Biologist doesn’t mind about the methods (as in paper)

• The blasting etc will just be there in a paragraph. If you find errors you re-submit correct versions of sequences.

• A lot of stuff looks similar as you work on it so a lab-book helps.• When is it useful? When writing up. Very useful when writing up

rather than wading through files. When have collaborators for presentations or talks. When you have a student you can just show them the process. Time- you can see if someone published a suggestion before you went ahead and did it. When you want to submit a sequence. When you have competitors. A collaborator did ask the date once – I could go back. In bioinformatics I’ve never had that.

Informal notes about lab-books

Research methods, EngD course, 2012 Slide 30

Key points

• Case studies are a retrospective analysis technique that examine a single case in depth

• Case studies rely on diverse information sources including (sometimes) fieldwork

• Fieldwork is a way of finding out about human and organisational factors relevant to a system

• Talk to users (stakeholders, developers, etc) and spend some time observing what they do

• Shut up, look, listen. They are the experts, not you.• Discuss your findings informally with other developers, and

write up your findings more formally if they are useful to a wide number of people