searching the internet what patent searchers should know

Eric Sieverts

WON, 11-12-2012

UB Utrecht HvA-MIC GO Opleidingen

agenda

• searching the web• the volatile google landscape• searchtools and functionality• google options• beyond google • beyond general web search

agenda

generalweb

search

specificmaterialsearch

importance of specificmaterialtypes?

the generalweb?=?

everything

how to … how to …

when& why

an ever changing google landscapean ever changing google landscape

• unreliable numbers • irreproducible results• disappearing functions• changing interfaces

"coping" with numbers of results

• expectation how combination of terms effects the number of results, are generally met in structured databases, but:

• with Google (and other web search) numbers are not stable, irreproducible, unreliable, with inexplicable effects– refining by adding AND-relation gives larger number, – expanding by adding OR-relation gives smaller number,– numbers are only quick extrapolation from part of search index,– depends on distribution of the index over servers,– depends on Google version, browser, whether logged in, history, ...– depends on Bing geographic setting

• Danny Sullivan explains why Google can not calculate: http://searchengineland.com/why-google-cant-count-results-properly-53559

Why Google Can’t Count Results Properly

Google as a vanishing engine

some services and options disappear completely– timeline, wonder wheel, toolbar, ...– + operator– real time results, code search – google buzz, google wave, ...

others are only hidden– links for advanced search and for settings hidden under “cog wheel”

(sometimes dependent on browser)– Scholar and Directory no longer mentioned in drop down menus– backlink search no longer in advanced search– search for "similar" pages & "cache"-link are hidden in "invisible"

pop-up page preview – …

like faceted search in for instance Scopus

refinements and additional functionslike in modern "web scale discovery" systems

but meanwhile

this is already

an "old" interface !

google.nl [until 2 weeks ago]

from clear left column facets & refinementsto blurry top menus (for mobile users?)

google.com

all options by material type, in old interface

search terms: Google is thinking for us

Google tries to improve and to broaden your queries• automatic spelling corrections (veilgheid >> veiligheid)• search for words with same word stem (singular/plural, verb,

conjugation, inflection, …)• expands acronyms (jfk >> john f kennedy | wwii >> world war II)• adds synonyms (vaccination >> immunization)• transforms separate words to compound term & vice versa (veiligheid

maatregel >> veiligheidsmaatregel | catfood >> cat food)• may leave out term as optional if not differentiating enough

more often and elaborate in English than in Dutch

• personalises search, based on previous search behaviour

and if you don't like all of this ........

never sure what/when or not

>> "verbatim"

new option introduced early 2012verbatim

on google.nl: "woord voor woord"

recentlymoved totop menu

standard semantic codingallowed Google to make arecipe search engine"embedded metadata"

standardisation of property descriptions in HTML

of recipe pages, with"microformats"/"rich snippets markup"

Google's "Knowledge Graph"knows 500 million objects with 3,5 billion properties(but only in English)

dates

??

no:

publication dates

• limitation while searching google– before search: only "past day/week/month/year"– after search: also limitation on custom range "from .. to .."

search tools:

publication dates

• limitation while searching google– before search: only "past day/week/month/year"– after search: also limitation on custom range "from .. to .."

• how reliable are google's dates? NOT• how else to determine date?

– look at page text (especially top and bottom or blogging date)– look in page source (HTML) for metadata– try entering javascript in browser URL bar

but does NOT work for CMS generated pages– try to find recent time stamped version in Web Archive

(waybackmachine)

javascript:alert(document.lastModified)

previous (disappeared) versions of pages

• recently disappeared: try search engine cache not just google! :

Bing

Yahoo

Exalead

previous (disappeared) versions of pages

• older versions: try web archive (waybackmachine)

• links within same site are mostly working

• if particular page has not been crawled, you get overview which other pages on the website have been crawled

reliability - general

general website assessment criteria• professional lay-out• indication of author/organisation (“about us”)• data about organisation: address, telephone, map/driving directions• indication of targeted audience• not too many advertisements and pop-ups (although every site has them)• clear navigation• internal search option• speed of web server• backlinks from well known organisations **• up to date-ness (with date given)• language use • interpret the URL/domain-name (eg: edu, edu.au, edu.sg, edu.ng, edu.lb, ac.uk,

gov, gov.uk, gov.hk, gov.au, gov.on.ca, gob.es, gob.mx, gob.ve, gob.ec, ...)

reliability - organisation

Information about organisation:• Google pagerank (backlinks)

use for instance: http://www.prchecker.info/http://www.checkpagerank.net/

• Alexa rank (web traffic)see for instance: http://www.alexa.com/

http://www.seomastering.com/alexa-rank-checker.php

• domain owner:use for instance: http://centralops.net/co/DomainDossier.aspx

http://whois.domaintools.com/

• search for "backlinks"

reliability - backlinks

search backlinks to particular web-page/-site • Google: link:http://www.domain.zz/folder/file.html

very incomplete result• Yahoo site explorer: died last year• DuckDuckGo: link:http://www.domain.zz/folder/file.html

no total number of results displayed• OpenSiteExplorer: linking pages + linking domains

very complete; also domain & page authoritypaid subscription for more than 3 queries a day

• Exalead: link:http://www.domain.zz/no backlinks to specific page, but to whole site

some more "how to" (google)

• domain search: site:edu OR site:edu.* [for all edu (sub)domains]

• url search: inurl:novelty

• title search: intitle:catalytic

• filetype search: filetype:pdffiletype:xls OR filetype:xlsxfiletype:doc OR filetype:docxfiletype:rss

• exact search: "greenhouses" [or VERBATIM for all words]

more than shown inadvanced search

drop-down

web search engines to try besides google• Bing (microsoft, large)• Yahoo! (content=Bing, large)• Exalead (french, many advanced functions, primarily demo system)• Blekko (uses hashtags to search more [domain-] selective

also many predefined hashtags; e.g. /likes for Facebook)• DuckDuckGo (assures privacy, no personalisation, no filter-bubble,

!Bang-function offers many extras, rather small)• Gigablast ("green" search engine, rather small, a few unique functions)• Millionshort (leaves out results from most popular sites the long tail)• Wolfram|Alpha (knowledge engine, facts, calculations)together, these others have 30% market share in US; in NL only 3%• Yandex (in Russia more popular than Google)• Baidu (in China more popular than Google)• Naver, Daum(in South Korea more popular than Google)• Seznam (in Czechia more popular than Google)

material type specific search

blogs google blogs, icerocket, technorati[rss] CTRLQ, RSS SearchHub

video google video, youtube, youtube edu channel, bing video, blinkx, voxalead-news

images google image, yahoo image, bing image, flickr,tineye (ip-check), panoramio (geo-search)

science google scholar, microsoft academic, scirus,oaister, scientific commons, science.gov

nieuws google news, yahoo news, bing news, cnn, bbc,historische kranten KB, historic american newspapers (LOC)

tweets twitter search, topsy, tweetzi, tweetscan, postpost, snapbird

social socialsearcher, socialmention, samepoint, whostalkin, kurrently

forums google groups, omgili, boardtracker

41

tweets & social search

• Twitter in 140 characters – often with shortened links– often with photo- or video-link – often with hashtags (#agreeduponkeyword)

search (often limited to last 1 - 2 weeks, and .... to those 140 characters)– twitter (also advanced search)– topsy, tweetzi, …– postpost (your own timeline - i.e. everything you're following)– snapbird (all tweets by 1 person - enter his/her twittername)– tweetscan (on limited scale also older messages)– twicsy (photo's on twitter) – ...

overview/review of tools: All the easiest ways to search old tweets

49

tweets en social search

• “Real time / social search engines”– socialsearcher, socialmention, samepoint, whostalkin, kurrently,

… (tweets + blogs + facebook + …)

– Google personal results / Google+ ("search plus your world") – real-time pictures: skylines

• Forum discussions– omgili, boardtracker, ...– Google groups (also old newsgroup discussions)

for research methods:– advice from Henk van Ess (dutch): "de digitale detective" (2012)– How to: use social media in newsgathering (2012)– 100+ Social Media Monitoring Tools (2010)