The Good, the Bad, and the Random:

An Eye-Tracking Study of Ad Quality in Web Search

Shuangze TangXu Tian

OutlineIntroductionRelated workMethodResultsConclusion

1. IntroductionSERP: Search Engine Result PageOrganic search Vs. Sponsored search

1. IntroductionGoal: How users distribute their visual

attention on different components of a SERP during Web search tasks.

2. Related workWeb Search Behavior in General The quality of the results and their presentation:-Using total time or overall search success (Turpin & Scholer @ SIGIR’06)-Search success is the same for both good and degraded system(Smith & Kantor @ SIGIR’08) The Search type- Three general classes of user goals: Informational, Navigational

and Transactional (Border et al. @SIGIR forum2002)- Searchers are more successful for common queries and

common goals (Downey er al. @CIKM’08) Individual differences- Search experts (White & Morris @SIGIR’07) and Domain experts

(White et al. @WSDM’09)

2. Related workEye Tracking on SERPs- Searchers examine a SERP is influenced by the position and

relevance of the results (Joachims et al. @ SIGIR’05; Pan et al. @ JCMC 2007; Guan& Cutrell @CHI’07 )

- Longer snippets lead to better search performance for information tasks (Gutrell & Guan @CHI’07)

- Two different types of searchers : exhaustive and economic (Aula et al. @INTERACT’05)

The Influence of Ads- Sponsored links (10% to 23% of all links) were presented on a

SERP (Hochstotter & Lewandowski @Inf. Sci. 2009)- Intergrating ads with the organic results does not increase their

click through rate (Jensen & Spink @ IJIMA 2009)

The InnovationFormer researches

mainly focus on how visual attention distributed on 10 organic results

However… additional element in the page

We are focusing on other components

The InnovationTask Type- Informational Vs. Navigational

Quality of ad:- Good Vs. Bad

Sequence of Good & Bad ads

3 MethodsInstrument: EYE-TRACKINGGaze-position: stand for visual attention. Tools:

Tobii EyeTracker & Tobii Studio

3Method IntroductionTop Ads

Organic Result

Search Engine Result Pages(SERP)

RightADs

3.1 Experimental design and Procedure

VariablesProcedure

Task VariablesTask type (informational/navigational)Quality of the ads (good/bad) shown on the

SERPsBlock (G/B/R) the trial belongs toCondition (GB/BG/RR) the participant was

assigned to

Task type example: Navigational Task(They had to find specific pages)Informational Task(They had to find factual information)

SERP Elements and SERP Generation

10 results Not containing any special

elements like maps, videos, images, or deep links

3 top ads 5 right rail ads Related searches 20 of the 32 initial queries

contained related searches

SERP Generation(con’d)Implement our own search interface show in

former slide. Initial task query, the interface showed a locally

cached version of the first SERP for the query.Sequent user-generated queries, the interface

queried a commercial Web search engine. A pool of good ads and of bad quality ads.

The static first SERPs for the initial task queries always contained the same ads from the appropriate pool. (Our focus)

For subsequent, ad from the appropriate pool were randomly selected and integrated into the SERP at runtime. (Not the focus in this experiment )

More detail:

Pool of GOOD Quality or BAD Quality Ads.

Initial Task querySame Ads

Sequent queryAds randomly and

integrated

Ad Quality The good ads Select from the ads shown by commercial web search engines such as Bing,

Google, and Yahoo, in response to the initial task queries.

The bad ads Select from the same commercial web search engines by generating queries

using a subset of the terms occurring in the initial task queries

Trail Sequences A trail is one unit of the experiment starting from reading the task description

until completing the task The experiment was divided into 4 blocks, of 8 consecutive trails Three types of blocks: Good(G), Bad(B) or Random (R) Each participant was assigned to one of 3 conditions GB (GBGB), BG (BGBG),

or RR (RRRR) The order of the tasks in a 32 trails sequences was randomly assigned

More in detail:3 types of Block

Good(G): 8 trails with mostly good ad quality.# of good ads > # of bad ads

Bad(B): 8 trails with mostly bad ad quality.# of good ads < # of bad ads

Random(R): 8 trails with good and bad ad quality equally.# of good ads = # of bad ads

The Experiment divided into 4 blocks, of eight consecutive trails.

To make the blocking effect less obvious to the participants, the ad quality in second trial of each G or B block is reversed.

(1)All ads: either good or bad //(2)Each participant was assigned to ONE of the 3 conditions GB, BG, RR.//(3)Each condition contains 4 blocks of trials GBGB, BGBG, and RRRR (16 g +16 b) //(4) The order of tasks: randomly assigned. //(5)Each unique tasks was performed in all 3 conditions//(6)The participants saw 32 trials without and special description of block structure and the quality of ads.

Summary of Independent Variables

The main independent variables for each trial:

Task type (informational/navigational)Quality of the ads (good/bad) shown on the

SERPsBlock (G/B/R) the trial belongs toCondition (GB/BG/RR) the participant was

assigned to

Tasks Description(1) To every participant, solve the SAME set of 32 search tasks.

Half are navigational( Find specific pages) Half are informational(Find factual information)

(2) Each task had a description telling the participants what they should do.

Provide them with an initial query for each task

(3) Cached results for each initial query. =>consistent initial set of result for each task (4) After the initial SERP was represented, participants were free to proceed as they wished.

ProcedureEach task, the participants with a written task

description and corresponding initial query. Reading the description and query aloud. Start first search query. (static page, locally stored)To solve the task, they need navigate to an

appropriate web page and point out the solution on it to the experimenter.

After finding solution, answer ”How good was the search engine for this task?”

Finished, fill in a study questionnaire asking about their WEB search experience and practices during the study.

Apparatus17” LCD monitor

1280x1024 pixels Browser Internet

Explorer 7 with a window size of 1040x996 pixels

Tobii x50 eye tracker (50 Hz )

software Tobii Studio.

Video sample

Participants 38 participants produced valid eye-tracking data

(out of 41), Recruited from a user study pool. Age (26 ---60) (mean = 45.5, σ = 8.2)Wide variety of backgrounds and professions. 21 female and 17 male. 13 unique task sequences.

13 participants assigned to the GB condition, 13 to BG 12 to RR.

Overall, we got valid eye-tracking data for 1210 trials.

3.4 Measures AOIs (Areas of Interest)All regions labeled in

right pictures are AOIs

3.4 Measures(cont’d)Fixation Impact fi(A)Function: determine the amount of gaze an AOI

A received. From: detected using built-in algorithms of Tobii

Studio. The algorithms generate a fixation if recorded gaze locations of at least 100 ms are close to each other. (35 pixel)

3.4 Measures(cont’d)Clicks c(A)Function: count the number of clicks on any links

on the SERPs. c(A) specifies the number of clicks aggregated

for the AOI A. e.g. The AOI top-ads containing all 3 top ads, c(top-ads) is the sum of clicks on any of the top ads.

Time on SEPR tFor each participant and task, the time on SERP

t measures the time the participant spent on the FIRST STATIC SERP for a give task.

t including time(all views of the first static SERP as well as the time to first click. )

3.4 Measures(cont’d)

4.RESULTGaze on the first static SERP represents 88% of

the total gaze on all SERPs

Focus on 3 aspects of gaze on SERPs: 1. Task type2. ads quality 3. orders of good/bad ads

4.1 General Gaze Distribution on SERPs

A gaze heat map describing the distribution of visual attention.

This picture the well-know gaze distribution referred to as “Golden triangle” and “F-shaped pattern.”

4.1 General Gaze Distribution on SERPs

The top ads receive as much attention as result found on the fold(at position 6,7)

Most Visual attention was devoted to the top few organic results.

4.1 General Gaze Distribution on SERPs• Gaze and click : in general agreement, some differences.

• There are more clicks that attention on top results• Top and right rail ads receive a higher fraction of visual

attention than of clicks • top and right rail ads receive a higher fraction of visual

attention than of clicks

4.2 Effects of Task Type

Average fixation impact for SERP elements, broken down separately for informational and navigational tasks.

4.3 Effects of Ad Quality

Average fixation impact for SERP elements, broken down separately for SERPs containing good and bad ads.

Twice visual attention to top ads when ads were of good quality

No reliable effects of ad

quality on this part

4.4 Sequence Effects

4.4 Sequence Effects(click)

The order of good/bad ad strongly affected the search behavior.

SERP has Random order of good/bad ads =>ignore

SERP has consistent Good ads => more attention

Predictability is an important factor

4.4 Discussion

4.5 Blocking EffectsDrop

>30%

5. conclusionthe top few organic result which is stronger for

informational than for navigational tasks. Furthermore, the quality of ads has a

significant influence on users search interaction. Good ads could attract more visual attention

Finally, gaze were strongly related to the order in which good and bad ads were presented. when ad quality varied randomly, participants are more likely ignore the ads, even though the ads were good on half of the trials.

Future workThis research represents a first step in understanding

how task, ad quality and sequence influence search interaction.

In our study, we focused on a specific static SERP composition which always consisted of 10 textual organic results with top and right rail ads.

next step:1. Add more SERP compositions, e.g., images, maps or

deep links2. In addition, we would like to explore how the quality of

ads interacts with the quality of organic results. 3. Finally, Develop more models of search processes and

strategies

Thanks!