HOW TO AVOID SEO’S BIGGEST DISASTERS

Mark Munroe SEORadar.com mark@seoradar.com @markemunroe #smx #23A

October, 2014

#smx #23A @markemunroe

SEO DISASTERS & REAL LIFE HORROR STORIES (AND HOW TO PREVENT THEM)

Warning, this presentation is not for the faint of heart. Please be prepared for the following: • Sudden drastic losses in traffic • Disappearing revenue •  Finger pointing •  The complete destruction of a

once exciting startup •  Lost jobs •  Zombies!

#smx #23A @markemunroe

The Laws • Murphy’s law “Whatever can go wrong, will go wrong” • Smith’s law “Murphy was an optimist!”

• Munroe’s law of SEO - “Anything that can break, will break. And things that cannot possibly break, will also break!!!”

#smx #23A @markemunroe

Summary of problems Problem Meta Robots noindex and/or nofollow

Noindex or nofollow on important conent

Rel- canonicals Pointing to the wrong place, syntax, improper use

Disappearing 301s Losing the value of historic links Disappearing on-site links Content drops from the index Titles, H1, content changes Removing important keywords Spammers Spam content, spam links Crawl issues Crawler spending time in the wrong paces

(faceted search). Nofollow links Nofollows to important content. 302s 301s magically turning into 302s Robots.txt Blocking content Unintentional cloaking Engineers trying to be too smart

#smx #23A @markemunroe

• Site accidentally places a no index on every page. •  Traffic drops to 50% before fixed. • Upon fixed, traffic never goes back to pre-problem level

and site loses 20% of traffic and revenue. • Potential exit for the site owner evaporates

True life disaster number 1

#smx #23A @markemunroe

How could that possibly happen? •  Think that can’t happen to you? Think again!

•  With conditional no-indexes proliferating now through a site, it’s easy for the logic to get swapped.

•  Templates with common code get moved, swapped. •  Logic that is intended for beta, white label and other duplicated

pages propagates to the live site. •  Example: “If BETA == True then insert_noindex” If a bug turns the variable BETA to “True” or if somehow ‘==‘ turns into “!==“ then the site gets noindexed.

#smx #23A @markemunroe

Robots.txt •  Yet another way to de-index your site or sections of your

site User-agent: * Disallow: / or User-agent: * Disallow: /blogs

#smx #23A @markemunroe

Here is a really creative way to de-index • On a major ecommerce site:

•  Somebody went into Webmaster Tools to delete a URL

•  They intended to delete a single URL •  Instead deleted a folder

#smx #23A @markemunroe

De-indexing - Identification and Prevention Symptoms

•  Sudden drop in traffic, over a short period of time as the noindex propagates. •  Impacts both Google and Bing, but not necessarily at exactly the same time. •  Even direct traffic may be be impacted as sometimes SEO bleeds into Direct

(misclassification). •  Drop in indexation

Prevention •  Check HTML source for meta robots noindex, nofollow tag whenever pages are

updated. •  Make sure to test all unique templates •  Verify robots.txt •  Create automated test scripts •  Use 3rd party audit software to help identify the problems •  If only Google or Bing impacted, look into webmaster tools for possible clues,

specifically removing folders.

#smx #23A @markemunroe

True life disaster #2 – the end of a promising startup •  This one happened in my early SEO career •  A steady decline appeared and the site lost ~around 40% of its traffic over a

6 week period. •  This came at a time that we needed more funding. •  With cash running out and in decline, site was sold for a song. Eventually,

resold to Groupon as part of a bigger sale, who then shut it down. •  At its peak, it had the most overlapping keywords with TripAdvisor according

to SEM Rush. •  How did this happen?

Site re-architecture required new URLs

301 www.site.abc www.site.xyz

•  301s makes sure that anyone clicking on the old link still get to the new page.

•  301s pass juice from the historic links that are distributed around the Internet.

•  So what’s the hairy, dark, creepy, insidious, monster hiding here?

#smx #23A @markemunroe

The 301 Risk 301 Cutover

•  In the early days, if the 301s are missing, they are easily identifiable. •  Everyone is focused on testing the redirects. •  If the 301s don’t work, it will be clear from search engine traffic.

What about 1, 2, 3, 6 months later? •  The truly insidious risk is that weeks or many months later, a simple

server config can break 301s. •  Nobody is watching anymore. •  Site functions fine. •  No problem reaching the site from search because old URLs are de-

indexed. •  Traffic can severely suffer because all the great historic links have been

severed.

#smx #23A @markemunroe

Identification and Prevention Symptoms

•  Broadly impacts all pages of the site as the entire site is weakened. •  Some top pages which had lots of links coming in may demonstrate

more significant drop. •  Increase in 404s in Webmaster Tools to old URLs (may lag the actual

problem by quite a bit).

Prevention •  Create scripts or use tools that verify the old URLs still redirect on a

regular basis. •  Tech education on the importance that the redirects persist over time. •  Periodic manual testing of the redirects. •  Periodic review of the 404 errors in Webmaster Tools to see if the old,

unsupported URLs show up.

#smx #23A @markemunroe

Missing Links or Changed Links •  Link structure of a large website is incredibly

complex and SEO is very much dependent on the linking structure.

•  UX re-design and product changes can remove links critical for SEO

•  Or bugs can cause links to disappear from

paged lists and/or HTML sitemaps.

•  On a site with millions of pages, it’s not easy to keep track.

#smx #23A @markemunroe

True Life Disaster # 3 – 9 months of lost traffic The Situation •  A company I joined had suffered a severe loss in traffic about 6

months earlier. •  They had concluded that the problem was related to Bing hitting

the servers too hard and slowing down Google’s crawl which resulted in increased time to crawl a page (hurting indexing).

•  My analysis found something very different.

#smx #23A @markemunroe

True life disaster # 3 – 9 months of lost traffic The Investigation •  I tried to identify changes that occurred during the time in

question. This was extremely difficult and release notes were poor.

•  I finally isolated the loss to a particular page type (long tail pages, a million plus).

•  I found only about 30% had a click-path. •  We then were to identify the specific bug that caused the

problem. •  Within 30 days of the fix, we started to regain traffic.

#smx #23A @markemunroe

Missing Link Path

Symptoms •  Gradual drop in traffic •  Gradual drop in indexing

Prevention •  Check source for meta robots noindex, nofollow tag whenever

pages are updated. •  Make sure to test all unique templates •  Create automated test scripts •  Use 3rd party software to help identify the problems •  If only Google or Bing impacted, look into webmaster tools for

possible clues, specifically removing folders.

#smx #23A @markemunroe

True Life Disaster # 4 – A little bit of knowledge is dangerous • At Trulia we acquired a site which had been on a long

downward trend. • On investigation, an engineer was trying to make page 1

rank better on a paged list and didn’t want the other pages to compete. So they nofollowed the paged links.

• Other links were removed to make it a ‘cleaner’ site. •  Oops - 6 months of traffic down 50% (and they blamed Panda).

All the paged links were

nofollowed.

#smx #23A @markemunroe

Missing Links/nofollowed links

Symptoms •  Gradual drop in traffic •  Gradual drop in indexing •  Impacts both Google and Bing (not necessarily in step) •  Only impacts a subset of content

Prevention •  Education so that everyone touching product knows the

importance of on-site links. •  Make sure to test all unique templates with a

Chrome plugin that highlights nofollow. •  Create automated test scripts or use 3rd party software that

alerts user to disappearing persistent links and nofollow links.

#smx #23A @markemunroe

True-life disaster # 5 Warning – this one is going to be really, really disturbing.

#smx #23A @markemunroe

True-life disaster # 5

•  Several years ago I was asked to help debug this ugly situation.

•  Clearly an algorithmic penalty which caused the site to bounce back and forth out of the index.

#smx #23A @markemunroe

Evil Spammers •  Spammers figured outs how to

use the blog to create landing pages.

•  Created massive amount of

pages focused on online drug sales (viagra, cialis and others).

•  Generated massive amounts of

links to these landing pages with rich anchor text.

•  Cleanup and recovery involved manually removing thousands of pages with Webmaster Tools.

#smx #23A @markemunroe

Spam penalty Symptoms •  Dramatic drop in traffic •  Inappropriate words showing up on your keyword list in Webmaster

Tools. •  UGC spam landing pages showing up your top pages report. •  Good news today is that you may get a UGC spam warning from

Google.

Prevention •  Build and train Bayesian filters to stop spam from getting posted. •  Moderation to identify and delete spam. •  Regularly examine keywords and landing pages in Webmaster

Tools. •  Periodically search site for likely spam words with site command.

#smx #23A @markemunroe

Quick summary of additional

disaster scenarios

#smx #23A @markemunroe

Rel Canonicals – the Fiat of tags •  Rel Canonicals – always breaking! •  Disappearing •  Broken Syntax •  Pointing to null •  Pointing to the home page •  Pointing to broken URLs •  Pointing the sites product cancellation

page (yes I saw that) •  Cloaked rel canonicals – (really, I couldn’t

make that up!)

Potentially very, very destructive - allowing all sorts of duplicate content to get indexed. We need scripts, people and tools testing them to make sure they don’t break.

#smx #23A @markemunroe

Accidental/intentional page changes with SEO impact SEO impacting page elements:

•  H1, H2, H3 •  Titles •  Alt Text •  Anchor text •  On page text

Prevention of issues: •  No Surprises

•  Training and education throughout the organization •  Automated Scripts, audit and test tools to identify problems •  Regular post release audit

#smx #23A @markemunroe

Mobile Mishaps • M.Dot issues

•  Broken canonicals on m.dot pointing back to desktop.

•  Broken rel alternate •  Serving incorrect version to mobile

spiders

• Regularly view pages and check source.

• Canonicals on m.dot is particularly important.

#smx #23A @markemunroe

Ajax and the incredible disappearing website •  Links in navigation • Search Results • Content • Navigation • Disappearing Pages

Ajax can work but must be implemented in a way that crawlers can find the content. Ultimately it can results all the content and link related problems I’ve already discussed.

#smx #23A @markemunroe

Disaster Avoidance Summary People •  Training •  Education •  Testing – regularly testing SEO impacting changes

Tools •  Custom Scripts •  SEORadar – designed specifically to identify and alert users for

potential SEO disasters (change analysis). Disclaimer: my company.

•  Audit Tools – ScreamingFrog, Raven, Moz, BrightEdge, SearchMetrics, Conductor