Exploratory search task performance and user satisfaction in the enterpriseEnterprise Search Europe, London 2015

Paul H. Cleverley

• Systems Thinking: A philosophical alterative to reductionism (pursuit of simple answers to complex issues). Fragmentation in order to make systems more manageable, risks losing sight of the big picture and consequences of actions taken (Senge 1990).– Single loop learning - operationalize actions

– Double loop learning - question the norms (Argyris and Schon 1978)

• Fallacy of centrality: Experts can overestimate the likelihood they wouldknow about a phenomenon if it was taking place. The “fallacy ofcentrality” (Weick 1995), i.e. because I don’t know about this event, itmust not be going on.

• Pragmatism: Concerned with warranted assertions, plausible explanationsthat fit experiences (not concerned with what is ‘true’)

Assumptions and Philosophies

• Two main types of search goal (Marchionini 2006)– Lookup/Known Item AND Exploratory search (Car Racing analogy)

• Exploratory searching is important – Missed business opportunities: 14% annual revenue (Oracle 2012)

– Poor search can miss evidence of fraud (Johnson, 2013)

– Has caused fatalities in the health sector (Savulescu & Spriggs, 2002)

• Search can be collaborative but is often an isolated activity

• Few causal models for exploratory search task performance

Background

Background – the car (the search engine)

Background – the car (the user interface)

Background – exploratory search UI’s

(Cleverley and Burnett 2015)

“.. with analogues you don’t know what

terms to query on, because you don’t know

what they are”

Background – Search Centre of Excellence

Background – Monitoring (search log analysis)

Techniques such as Failed Search Analysis and Ranking tests such as Mean Reciprocal Rank (MRR) can be biased towards Lookup/known item search goals

Background – engine oil (the content)

Background – focus on the searcher (the driver)

• Search Literacy

– “many believe a position has been reached that professional education and research are irrelevant to practice” (Wilson 2008), “We need more ..study of the unique needs and challenges of increasing information literacy skills in the workplace.” (Abram 2013)

– Increasing search literacy may mitigate information overload (Bawden 2009)

• User satisfaction

– “need to study the relationships ..between various user, environment characteristics and satisfaction.” (Al-Maskari & Sanderson 2010)

• Task performance

– Where a ‘gold standard’ is used (TREC) studies do not measure how the searcher felt or how they (or organization) may have reacted to performance feedback, “we found..very few studies attempted to use some objective form of measuring the level of user’s search performance.” (Moore et al 2007)

Background – focus on the searcher (driver)

• H1: There is a difference in user satisfaction (overload to non-overload)

• H2: There is a difference is search task performance (overload to non-overload)

• H3: There is an association between user satisfaction and search task performance

• H4: There is an association between self reported search expertise and search task performance

• Q1: What praxes and traits lead to search task success?

• Q2: What are the underlying antecedents for search task performance?

Research Questions

Methodology – Mixed Methods

Use of the companyenterprise search engine1. Use search box only2. 10 minutes per task3. Find 10 most relevant items

Two search tasks, Task1 providing a feeling of information overload (>500), Task2 a limited amount of possible relevant results (<100)

Influence of subject matter knowledge mitigated by task design, choices made just on metadataSearch Task 1—Gather recent gravity, magnetics reports for Peru (Task2 same but for Cyprus)

4 high value items were added by the researchers for each task

Case study – oil and gas company

Permission frommanagement

26 experienced (>10 years)Information Managementprofessionals supporting theexploration department

Methodology – Mixed Methods

Participants provideUser satisfactionfor Task1 and Task2 using a Likert scale

Participants provide 10 ‘most relevant’ items each for Task1 and Task2

Researchers identified how many of the high value items were found per participant which was used as the task performance measure

The participants were unaware the researcher was able to view the search log in real time

Task results (performance and patterns from the search log) were fed back to participants and management in interviews. Questionnaires collected more data

Methodology – Mixed Methods

Qualitative: Thematically mapinterview data using an approach based on grounded theory

Quantitative: Statistical inference tests:Kruskal-Wallis, Mann-Whitney U Wilcoxon Signed RankSpearman Rank CC

Task order, age, gender, native language, personality effects tested

Co

ncl

usi

on

sTriangulateConvergence Coding Matrix (CCM)

Share results with other organizations in interviews

RESULTS: H1 User Satisfaction (US) difference

54% satisfied for Task1 (Information overload), 65% Task2 (limited results)No statistical significant difference

RESULTS: H2 difference in task performance

18% high value items found for Task1 (Information overload), 36% Task2 (limited results). Difference is statistically significant. Overall performance considered poor.

RESULTS: H3 association US & task performance

No association between task1 (overload) and user satisfaction. There is a statistically significant association between task 2 (limited) and user satisfaction

RESULTS – NEW THEORY PROPOSED

A statistically significant association exists between user satisfaction ‘delta’ (between Task2-Task1) and search task performance. Mental models play a role.

SE (USL-USO)

Relative User SatisfactionTheory (RST)

Where:SE=Search ExpertiseUSL= User satisfaction (Limited results)USO= User satisfaction (Overload)

RESULTS: H4 self reported search expertise

There is no statistically significant association between self reported searchexpertise and search task performance

RESULTS: Q1 Praxes and traits that led to success

As well as knowledge of search query construction, Metacognition“thinking about thinking” is likely to be a key factor in search task performance

1. Absorbing task instructions

2. Recognising implications of only using plurals

3. Query discipline and remembering

4. Avoiding Boolean OR queries

5. Using wildcards correctly

6. Bruce force persistence

7. Creativity

8. Effective results synthesis

9. Adaptation (learning from results returned)

• In case study organization (surprise)– The searchers

• “Unbelievable” [P19], “Interesting” [P6], “Very useful” [P21], “I obviously need to experiment more in the searches” [P19], “I will do things differently next time!” [P25].

– The organization: • General Manager for Global Exploration IM “It’s very surprising in 2015,

that something so trivial [B2] is not handled as standard by all search engines.”

• Search CoE Manager: “We are responsible for making the enterprise search engine work and that people can use it, not whether people are capable of knowing how to search.”

RESULTS: Feedback Interviews

• In external organizations (Transferability of findings)– CEO Professional Society providing information “Interesting. I would probably

rate myself as very good, but it would not surprise me if I turned out to be very poor!” [O1].

– Knowledge Manager Defence Sector organization, “Nobody takes a strategic overview of search other than making sure the IT service works” [O3]

– Aerospace Search CoE Manager ”Very interesting for me to see, in the end, search in big enterprises is looking at the same type of challenges.” [O5].

– Interviews with six organizations (in aerospace, pharmaceuticals, defence and oil & gas sectors) indicate the expertise of searchers is not monitored or fed back.

RESULTS: Feedback Interviews

Q2: Improve search: Do we need to learn to learn?

TACIT

EXPLICIT

Individual Experiential

Learning(Kolb 1984)

OrganizationalLearning

(Argyris & Schon 1978)

INFORMAL FORMAL

e.g. Enhance using

User Interface Scaffolding

e.g. “Bottom up” Integrating social

networks more closely with enterprise search

user interfaces

e.g. “Top down”Health check

experiments in high leverage/risk areas to assess search literacy.

• In information overload environments, there may be no association between user satisfaction and actual search task performance for exploratory search.

• The research data in this sample suggests searchers may not be able to self assess their search competence accurately.

• Based on the experience of participants, both they and the organization (management) were surprised at their low performance. In could be inferred that a lack of effective ‘learning’ loops has caused this.

• This may have been caused by management bias towards technology, single loop learning and a lack of ‘systems thinking’ and double loop learning with respect to search capability in the enterprise.

Conclusions

Thankyou for listening

Cleverley, P.H., Burnett, S., Muir, L. (2015). Exploratory information searching in the enterprise: A study of user satisfaction and task performance. Journal of the association for

information science and technology (JASIST)http://onlinelibrary.wiley.com/doi/10.1002/asi.23595/abstract

Blog: www.paulhcleverley.comemail: p.h.cleverley@rgu.ac.uk