“All databases are equal... …but some are more equal than others.” Stephen Adams, Magister...

transcript

“All databases are equal...

…but some are more equal than others.”

Stephen Adams,

Magister Ltd., GB

Topics

• Where database creation goes wrong…

• Why bother to evaluate?– A word about ‘quality’

• Quality content– missing documents, document kinds and

fields

• Quality context– search engines

• Conclusion

Topics

fields

• Conclusion

The basics of information retrieval

Documents Documentrepresentation

Query representation

HitsMatching

Adapted from Crestani, J.Inf.Sci. 29(2), 87-96 (2003)

Reference interview

HitsMatching

“I’m sorry - I don’t understand the question…”

“Are you also interested in…?”

“How much do you already know about this?”

QUALITY RESULTS START WITH US.

Strategy development

HitsMatching“Where has that manual got to…!”

“When did they start using that field?”

“Is that field available for all records?”

Document quality - at source

HitsMatching“I leave the form-filling to the paralegals…”

“I’m sure my secretary never transposes application numbers - she can read my handwriting…”

“Our patent office uses that INID code differently…”

Full text, abstract, indexing...

HitsMatching

“I get so much rubbish with full-text…”

“I don’t trust abstracts - especially for a freedom-to-operate search…”

“Their timeliness has improved - but indexing quality is down…”

“800,000 corrections per year”

Hitting the keyboard

HitsMatching

“Where on earth did that false drop come from…?”

“We always use the free services - the results are OK so far”

“Why does this host always crash on a Friday?”

Major topics for today

HitsMatching

Database content

Database context

Content and context

• The effectiveness of “a database” as a search tool is a function of (at least) two variables:– the data content– the search engine / command language.

• The ideal answer may be a compromise:– (‘average’ database & ‘good’ command

language) or (‘good’ database & ‘poor’ search engine).

Topics

fields

• Conclusion

Why evaluate database content?

• Database evaluation is a basic part of information literacy:– “a set of abilities requiring individuals to

recognise when information is needed and have the ability to locate, evaluate and use effectively the needed information.”

– American Library Association 1989, Final Report of the ALA Presidential Committee on Information Literacy

• If we do not evaluate our sources, we cannot serve our customers fully.

The biggest database of all?

Isn’t that enough for anyone?

So why evaluate?

A simple evaluation parameter: language

Cyber Atlas distribution 2000

English

Japanese

German

Chinese

French

Spanish

Russian

Source: CyberAtlas, www.clickz.com/stats/big_picture/demographics/article.php/5901_408521

OCLC figures for 2004 are comparable: 30-35% of the Internet is not in English.

Implication:

• The effectiveness of ‘the Internet’ as a retrieval tool will be skewed according to the nature of our search:– “Hermann Hesse” “Das Glasperlenspiel” =

13,600• of which “& domain=de” comprises 13,400

– “Hermann Hesse” “The Glass Bead Game” = 12,500

• of which “& domain=de” comprises 128

– “Hermann Hesse” “Magister Ludi” = 5,100

The third leg

• Good database evaluation should include not only the 2 factors identified above: – Database content i.e. how well it is put together

– Database context i.e. the command language and search engine

• but also a third factor– How well does this database fit my specific enquiry? (one-off

need or recurring usage)

– Note - if the evaluation process includes this factor, it follows that there is no such thing as the ‘ideal’ database for all enquiries

What is quality?

• “Fitness for purpose”– content– completeness– timeliness etc.

• It is difficult to be absolute; more easy to evalutate as a relative quantity– benchmarking two sources against one

another gives a better practical feel for ‘quality’ than attempts to measure against a mythical standard

Simple example of quality

• We wish to conduct a freedom-to-operate search in respect of Germany– one file contains DE-C2, DE-B4 documents– a second file contains DE-C2, DE-B4, DE-

C1, DE-B3, DE-T2 and DE-U documents

• Which one would you choose?– Whichever your answer, it does not imply

that the other is ‘poor quality’.

Measuring quality

• We can measure good content– essentially quantitative, binary

• We can measure good database structure/context – essentially qualitative, relative, subjective

• e.g. are there explicit links between individual records (e.g. common indexing scheme)?

• e.g. do the command language features or field standardisation facilitate virtual links?

• e.g. what proportion of the time is the system up?

The coelecanth

Location: GreenlandZone: polarHabitat: fresh waterSize: 30 cm.Era: 200 m. years agoExtinct for 50 m. years

Location: South AfricaZone: sub-tropicalHabitat: salt waterSize: 1.75 metresEra: 1938Alive and breeding

Databases or datadumps?

• Science is not ephemeral - it is cumulative– Unless adequate consideration is given to the

issue of retrieval at a distance of 10, 20 or 50 years after publication, then the resulting file is not a database at all - it is a datadump

• Much emphasis has been given in recent years to timeliness i.e. adding new records– add in haste, repent at leisure?

Robert Maxwell:

Chairman of Pergamon Press

Owner of Pergamon Orbit-InfoLine

Owner of Mirror Group Newspapers

“All the science that’s fit to print”

• Publication or ‘laid open to public inspection’ without consideration of retrieval afterwards means that each record is left isolated from the context of the corpus of science– and will be missed in a proportion of the

searches to which it is a relevant answer– or possibly never found again

Topics

fields

• Conclusion

Missing fields

Three ‘layers of incompleteness’

Missing Kinds-of-documents

Missing documents

• The classical measure of database quality:– Is

• every document of the same kind,

• published in that period

• by that publishing authority

– present in the file?

• Examples:– Latipat, USPTO.gov, Patent Abstracts of

Missing documents

• Latipat– Newly launched esp@cenet portal,

http://lp.espacenet.com

• USPTO.gov– Full-text of granted patents

• Patent Abstracts of Japan– JAPIO file

Latipat

10001500

200025003000

35004000

45005000

Both Latipat and PlusPat (below) suffer from the same problem - missing records; lots of them!

USPTO.gov

Partial listing of missing patents:

4097518 - 4097928 (411)

4526401 - 4527286 (886)…

= 6,092 missing between 4,000,000 and 4,999,999 (0.6%)

STILL 224 missing between 6,000,000 and 6,101,209 (0.2%)

PAJ fact sheet from Questel-Orbit

What the publicity impliesA

TECHNOLOGY

First limitation - by applicantA

TECHNOLOGY

1976 1989

Backfile to 1989 now available - but has every host loaded it?

Prior to 1998, cases not claiming JP priority were not automatically included in PAJ

Second limitation - by technologyA

TECHNOLOGY

1976 1989

Prior to 1989, only 48 out of 118 IPC classes were covered completely (40%)

Complete IPC coverage from 1989 - but no plans to create back-file?

The (messy) truthA

TECHNOLOGY

1976 1989

How to evaluate?

• “Missing documents” is one of the few parameters which can be measured independently of the database– Annual Reports of the office concerned– WIPO Industrial Property statistics

• Caution : – these may not refer to the appropriate

document kinds; check before use.

Caution

• Determining database ‘completeness’ is only meaningful when measured against a quantitative parameter– e.g. publication number.

• It has little or no meaning when measured using more qualitative parameters– e.g. no. of hits found using the same strategy

across several databases• the strategy will be sub-optimal for some

databases and not for others

Simple source-by-source comparison

BIOSIS -v- Medline

BIOSIS Evolutions vol.9 no.6 © BIOSIS

Science Direct: Comprehensive - provided it’s from Elsevier...

Web of Science: Comprehensive - provided it’s got a high impact factor from ISI...

MDL: PCT and EP from 1976 ?

Take-home message

• There is nothing wrong with publicity– provided it is not confused with user

documentation.

• Database producers still have a long way to go in informing users of the gaps in their databases– it should be much easier to locate this data

than it is at present.

Topics

fields

• Conclusion

Missing Kinds-of-Documents

• Second measure of database quality– Is

• every document of every appropriate kind,

– present in the file?

• Examples:– Overlapping year / country coverage– EP-A1, -A2, -A3, -A8, -A9– US-B1, -B2, -E, -C1, -C2

But they all cover Australia...

• Even given overlapping country and year coverage, different sources can cover different publication stages

• e.g. Australia– WPI : AU-A from 1963, AU-B from 1993– INPD : AU-A from 1973, AU-B from 1978– CAS : AU-B from 1927

• AU-A is included in CAPlus family, even though it will never be selected as CAS basic - see http://www.cas.org/EO/patkind.html

European correction documents

• ST.50 implemented from 1997– how many database producers take the data?– how many tell their users whether they take

the data?

• Examples:– Questel-Orbit EPPATENT file– STN Europatfull file

Coverage of correction documents

1/1 EPPATENT - (C) Questel.Orbit- imageCPIMPN - EP954211 A2 19991103 [EP-954211]BPN - 1999-44ET - Supporting apparatusBRR - 2000-29 (Updated 2000-29)DREX- 2001-01-18 Request for examination (Updated 2001-13)DNEX- 2001-08-06 First examination report (Updated 2001-38)DGR - 2003-07-23 Grant (Updated 2003-30)BGR - 2003-30 (Updated 2003-30)NGR - B1 (Updated 2003-30)

EPPATENT MAX format (edited) : all Bulletin announcements

Coverage of correction documents

L1 ANSWER 1 OF 1 EUROPATFULL COPYRIGHT 2004 WILA on STN PATENT APPLICATION - PATENTANMELDUNG - DEMANDE DE BREVET AN 954211 EUROPATFULL ED 19991114 EW 199944 FS OSTIEN Supporting apparatus.PIT EPA2 EUROPAEISCHE PATENTANMELDUNG GRANTED PATENT - ERTEILTES PATENT - BREVET DELIVRE AN 954211 EUROPATFULL UP 20030729 EW 200330 FS PSTIEN Supporting apparatus.PIT EPB1 EUROPAEISCHE PATENTSCHRIFT

Europatfull default format (edited) : no record of anything after EP-B1

INID (15) shows that this B8 is a correction to the B1 (grant).

How effective is this?

• The experienced information specialist is tempted to infer legal status information from the presence/absence of a particular publication stage (risky!) – e.g. EP-B = assumption of entry into force

• The inexperienced information specialist is not always given the correct links to lead to the right conclusion e.g.– e.g. US parent, re-issue, re-examination cases

Re-examination mentioned in facsimile version - but not in ASCII text:Parent case - claims 1-10Re-exam 1 - new claims 11-112Re-exam 2 - new claims 113-126

IFI record consolidates all changes into a single record - the novice has a better chance of getting a more accurate answer to a legal status search.

US coverage

Kind Code DefinitionEarliest date of use

Dialog / 652-654 IFI Claims INPADOC (incl. Delphion)

Questel / USAPPS

Questel / USPAT

STN/ USPAT2

STN/ USPATFULL WPI

US-A old Act grant 1836 1971 1950 1968 1971 1971 1963new Act published application 2001 2001 2001 2001 2001 2001

US-B old Act re-examination Y 1981new Act grant 2001 2001 Y

US-C new Act re-examination 2001 YUS-E re-issue 1838 Y 1963 1968 Y 1970US-H defensive publication 1969 Y 1963 1977 1976US-H1 Statutory Invention Registration 1985 Y 1963 1985 1985 Y 1968US-A1 Trial Voluntary Protest Program 1975 1975US-S Design Patent 1843 Y 1976 2001 1976 YUS-P old Act Plant Patent 1931 Y 1976 1994 1976 YUS-P1 Plant Patent published application 2001 2001US-P2 Plant Patent grant 2001US-A0 NTIS invention applications 1974 1983US-A9 Correction of new Act published application 2002 2002 2002None Office of the Alien Property Custodian (APC) 1917

• Example analysis of KD coverage– e.g. IFI would appear to cover SIR’s from 1963,

some 22 years before they started (?)

– e.g. split between USPATFULL/USPAT2 difficult to discern

Topics

fields

• Conclusion

Missing fields

• Third measure of database quality• every document of every appropriate kind

• present in the file to the same level of detail

• Evaluation must compare like-with-like• Variations in completeness of coverage and/or field

population will affect the apparent effectiveness

• Examples: • EP and PCT full-text files, Derwent WPI

coverage

Non-systematic or missing fields

• New field during database life– imposes an implicit time range on your

search• e.g. IPC editions, WPI coding changes

• Systematic omission of a field– biases results against records which do not

contain that field• e.g. US-A assignees

• e.g. JP, CN inventors in WPI

European Patents Fulltext covers all European patent applications and granted European patents published since the opening of the European Patent Office (EPO) in 1978…

But…EP-A specifications from 1986 in only one languageEP-B specifications from 1991 in three languages

PCT full text

• Many files claim to cover ‘full text’ PCTs– Few handle the cases published in Japanese,

Chinese or Russian• but these still have an English abstract

• Abstract searching gives equal weight to all documents

• Full text searching skews results in favour of records containing full text

Derwent WPI countries

• Most countries in WPI are coded using the Manual Code system– but not all countries had Manual Codes added

from the start of their coverage

• A strategy incorporating Manual Codes imposes an implicit time ranging on some countries, and can distort retrieval– MC retrieval of KR-B started 1990, biblio

available from 1986

Topics

fields

• Conclusion

Quality and the search platform

• A poor search platform / command language can ruin a good quality database, by effectively concealing or distorting the information which is present.

• Typical questions:– does the default print format contain the most

useful information for my search?– do I obtain the same answer irrespective of

the route to it?

Default print formats

1/1 PLUSPAT - (C) QUESTEL-ORBIT- image CPIM (C) Questel-Orbit PN - EP0954211 A2 19991103 [EP-954211] TI - (A2) Supporting apparatusSTG - (A2) Pub. Of applic. Without search report

1/1 PLUSPAT - (C) QUESTEL-ORBIT- imageCPIM (C) Questel-OrbitPN - EP0954211 A2 19991103 [EP-954211]PN2 - EP0954211 A3 20000719 [EP-954211]PN3 - EP0954211 B1 20030723 [EP-954211]PN4 - EP0954211 B8 20040414 [EP-954211]TI - (A2) Supporting apparatusSTG - (A2) Pub. Of applic. Without search reportSTG2- (A3) Publi. Of search reportSTG3- (B1) PatentSTG4- (B8) Modified first page

PlusPat BIB format : only shows first publication stage.

PlusPat MAX format

Variation due to search route

• US patent term extension under 35 USC §136 (Hatch/Waxman)– issued in the form of a Certificate of

Correction

• At least two equivalent routes to view: – locate the original document and check for a

‘Correction’ segment in full text view OR– go directly to list of term extensions and link

to Certificate• http://www.uspto.gov/web/offices/pac/dapp/opla/term/156.html

Test question

• Is there an extension in force for – US 4540568 ? – US 4572909 ?– N.B. - This question avoids the use of PAIR

(inoperative on day of test) and assumes that the enquirer has already established that US 4540568 has been replaced by US Re 32969

• but why was PAIR not working?

• and why should I have to make that assumption?

35 USC 156 listing

Results

• US Re 32,969 (replaced US 4540568)– via 156 listing, obtains a PDF of Cert. of Correction

shows term extended for 931 days • actual extension in listing is recalculated as 897 under 35

USC 156(c)(3)

– via full text view, there is no record of the Certificate of Correction at all; nor any link from US 4540568

• US 4572909– via 156 listing, obtains a PDF showing extension for

1252 days

– via full text view, an additional ‘Correction’ segment available

Additional document segment is present for US 4572909, but missing for others….

Summary answer

Source: US 4540568 /US Re32969

US 4572909

35 USC 156listing

Yes Yes

Full text No Yes

PAIR ? ?

Topics

fields

• Conclusion

Conclusion

• There is no such thing as “the database for all seasons”

• Evaluation is ongoing, even for established products

• There are many ways in which databases can be ‘incomplete’

• A poor search environment can ruin a good database

• Communication between legal, information and database specialists is the key quality factor

Coming up in part 2...

• Two case studies– PCT publication rates– Searching for gold

Enjoy your break!