ISI Research Services Group:

ISI Research Services Group:

Analytical tools for Research Evaluation

ISI Research Services Group Nancy K. Bayers

Outline

• Research evaluation: – Unique features of Web of Knowledge

– Journal Selection

– Quantitative tools

• Cui bono: What sectors use these analyses

• Citation Analysis – Issues and Caveats

• Free Stuff!!


The Web of Knowledge is…

• An integrated, web-based environment providing high quality content and the tools to access, analyze, and manage scientific and scholarly information

• Incorporating new technologies (linking, searching, and personalization) to integrate, extend, and enhance the research environment


E-Journal Full-Text

E-Patent Full-Text

Evaluated, Selected Web

Content

Web of Knowledgesm

A Free Resource!!


A. Journal Selection Process

• “A cost effective index index must restrict its

coverage, as nearly as possible to only those

items that researchers are likely to find useful.”

-Eugene Garfield, Ph. D.

ISI Web of Knowledge


… Journal Selection

1. Basic Journal Publishing Standards

2. Editorial Content

3. International Diversity

4. Citation Analysis




… Journal Selection

1.) Publishing Standards

a.) Timeliness

b.) Editorial Conventions

c.) English Language Bibliographic Information

d.) Peer Review


... Journal Selection

1.) Basic Journal Publishing Standards

a.) Timeliness of Publication

• A journal must be published according to its stated publication schedule to be considered for coverage in the ISI database.

• ISI must receive three on-time issues in sequence.




1. Basic Journal Publishing Standardsa.) Timeliness of Publicationb.) International Editorial Conventions

To optimize retrievability:- Informative Journal Titles

- Descriptive Article Titles- Full Address Information for Every Author- Complete Bibliographic Information for All Cited References




1.) Basic Journal Publishing Standards a.) Timeliness of Publication b.) International Editorial Conventions c.) English Language Bibliographic Information

- Article Titles- Author Names and Addresses- Cited References- Abstracts and Keywords





1.) Basic Journal Publishing Standards a. Timeliness of Publication b. International Editorial Conventions c. English Language Bibliographic Information d. Peer Review Application of the peer review process is another indication of the journal’s standards, the overall quality of research presented, and the completeness of cited references.



2.) Editorial Content

• Do we need this new journal in the database, or is the category already well covered?

• Does this journal offer something new?




3.) International Diversity

• Are the authors, editors, and editorial advisory board members from around the world?

• Does this journal reflect the global context in which scientific research takes place?




...3. International Diversity

Regional Journals:

To properly reflect the global context in which scientific research takes place, and to provide balanced coverage in each category, ISI also seeks to cover the best regional journals as well.




4. Citation Analysis

• Expert use of citation data help ISI identify important, influential, and useful publications.

• ISI measures use by analyzing citation data.




... 4. Citation Analysis

• New Journals

Has the past work of authors and editorial advisory board members received citations?

• Established Journals

Impact Factor



... Journal Selection ...4. Citation Analysis

Impact Factor: The average number of times recent articles

in a journal were cited in a particular year.

Nature Genetics:

2000 Impact factor: 30.910


Journal Citation Reports (JCR) -- 2001 Science EditionImpact Factor Calculation

Journal:Nature Genetics

Impact Factor: 29.600

Cites in 2001 to articles published in : 2000 = 6,1671999 = 7,538

00+99 = 13,705

Number of articles published in: 2000 = 2531999 = 210

00+99 = 463Calculation:

Cites to recent articles 13,705 = 29.600Number of recent articles 463



Research Evaluation

• Web of Knowledge: unique features–cited references -- footnotes–all author names–all author addresses

• Count –papers -- output–citations -- influence or impact

• Baselines


Cui bono?

• Policymakers

• Research directors & administrators

• Researchers

• Publishers

• Information specialists


Words to the wise...

• Citation analysis is a fair evaluative tool in those fields where publication in the journal literature is the main vehicle of communication.

• Need multiple indicators. Citation analysis is only a quantitative

supplement to other measures of research evaluation,

including peer review

• There are always exceptions to everything.


Citation analysis: issues and caveats

• Field publication behavior differences

• How to handle...– Self cites and negative cites– Reviews and methods papers– English language bias

• Skewed distributions

• Relative measures/baselines

• Confidence levels

• What citations do not reflect


Field citation rates: average cites 1997-2001

• Cell & Developmental Biology 15.9

• Molecular Biology 10.6

• Biochemistry & Biophysics 9.1

• Neuroscience 7.3

• Chemistry & Analysis 5.9

• Physics 4.4

• Earth Science 3.5

• Materials Science 2.1

• Mathematics 1.3


Field publication rates/size

• Clinical Medicine 24%

• Chemistry 14%

• Physics 12%

• Biology & Biochemistry 8%

• Plant & Animal Science 6%

• Materials Science 4%

• Neuroscience 4%

• Geosciences 3%

• Mathematics 2%

• Computer Science 1%

Field % papers in world


Self-cites and negative cites

SELF CITES

• Cumulative nature of individual research

• a natural and acceptable procedure if not excessive

• Approximately 25% in biomedicine

• 16% in plant physiology

• 17.5% in neurobiology

NEGATIVE CITES

• Only 7%


Reviews and methods papers

• Highly cited

• Influence subsequent research

• Lay foundation


English language bias

• English is the language of science

• Representative journals from

each region and nation

• Not significant for most analyses


Skewed distribution in bibliometrics

• 80/20 rule

• 20% of papers get 80% of cites

• 50% of papers are never cited


Relative measures--baselines

• Need context -- what does X cites mean?

• Normalize by size -- average citation rates

• Normalize by field -- average field citation rates


What citations to not reflect

• Clinical importance

• Public policy influence

• Regional or local interest


Confidence levels

Large data sets (nations, regions, large institutions)

Engineering & Applied Sciences

Fundamental &Basic Sciences

Small data sets(small departments, individual researchers)

high

low moderate

moderate


In-Cites: www.in-cites.com


In-Cites: www.in-cites.comEditorial Commentary by Highly Cited Scientists


In-Cites: www.in-cites.comHot Paper Commentary in Medicine


Special Topics: www.esi-topics.com








Science Watch: www.sciencewatch.com

ISI Research Services Group:Analytical tools for Research Evaluation

Thank You!!

Date post:	06-Jan-2016
Category:	Documents
Upload:	dympna
View:	34 times
Download:	0 times

ISI Research Services Group:

Documents