The Evaluation of Ontologies

Editorial Review vs. Democratic Ranking

Barry Smith

1

The need for ontology evaluation

Ontologies are expensive.

$8 million have been invested in the Gene Ontology thus far

Has this investment been worthwhile?

Are some ontologies more useful than others?

What are ontologies useful for ?2

Uses of ‘ontology’ in PubMed abstracts

3

In the olden days

people measured lengths using inches, ulnas, perches, king’s feet, Swiss feet, kanejaku, shaku, whale shaku, etc., etc.

5

then, on June 22, 1799,everything changed

we now have the International System of Units

Through the SI System

science becomes a cumulative, distributed endeavormy measuring equipment can be callibrated against your measuring equipmentmy hypotheses can be checked against your data

When should a new unit should be included in the SI system?

The work of the

CIPM International Committee for Weights and Measures

rests on an editorial process

9

Obvious benefits of a peer review process for scientific work

Creating an environment which rewards better work

Helping people to find better work

...

10

Proposal: Evaluate ontologies via an editorial process

of the sort used for scientific journals and scientific research projects

a process of peer review by human experts

11

Peer review process

• appropriate when evaluating science

• (not, e.g., when evaluating poetry, or fairy tales, or Chinese mythology ...)

12

Peer review process

appropriate in the domain of biomedical ontology

– ontology evaluation here can be of particularly acute concern

13

14

RELATION TO TIME

GRANULARITY

CONTINUANT OCCURRENT

INDEPENDENT DEPENDENT

ORGAN ANDORGANISM

Organism(NCBI

Taxonomy)

Anatomical Entity(FMA, CARO)

OrganFunction

(FMP, CPRO) Phenotypic

Quality(PaTO)

Biological Process

(GO)CELL AND CELLULAR

COMPONENT

Cell(CL)

Cellular Compone

nt(FMA, GO)

Cellular Function

(GO)

MOLECULEMolecule

(ChEBI, SO,RnaO, PrO)

Molecular Function(GO)

Molecular Process

(GO)

The OBO Foundry

THE OBO FOUNDRY

15

a suite of reference ontologies in the biomedical domain satisfying certain basic criteria and subject to an on-going process of peer review

OBO FOUNDRY CRITERIA

16

The ontology is open and available to be used by all.

The ontology is in, or can be instantiated in, a common formal language.

The developers of the ontology agree in advance to collaborate with developers of other OBO Foundry ontology where domains overlap.

17

CRITERIA UPDATE: The developers of each ontology

commit to its maintenance in light of scientific advance, and to soliciting community feedback for its improvement.

ORTHOGONALITY: They commit to working with other Foundry members to ensure that, for any particular domain, there is community convergence on a single controlled vocabulary.

18

CRITERIA

IDENTIFIERS: The ontology possesses a unique identifier space within OBO.

VERSIONING: The ontology provider has procedures for identifying distinct successive versions.

The ontology includes textual definitions for all terms.

CRITERIA

19

CLEARLY BOUNDED: The ontology has a clearly specified and clearly delineated content.

DOCUMENTATION: The ontology is well-documented.

USERS: The ontology has a plurality of independent users.

CRITERIA

20

COMMON ARCHITECTURE: The ontology uses relations which are unambiguously defined following the pattern of definitions laid down in the OBO Relation Ontology

CRITERIA

The OBO Foundry provides guidelines (traffic laws) to new groups of ontology developers in ways which can

• help to ensure interoperability through prospective synchronization

• counteract dispersion of effort • prevent some common types of

nonsense21

22

Example: the Foundry seeks orthogonality

This brings division of labor and other benefits

Foundry editors adjudicate in areas of overlap

How the editorial process works

23

Foundry editors balance– the flexibility that is indispensable to

scientific advance

– the institution of principles that is indispensable to successful coordination

How the editorial process works

24

is a top down approach, relying on authority

Peer review

25

democratic ranking

An alternative, bottom up approach

26

If we build it, will they come? Social engineering of new technology to

disseminate biomedical ontologies

Mark A. Musen and the BioPortal Team

Stanford University

27

If we build it, will they come? Social engineering of new technology to

disseminate biomedical ontologies

presentation to Ontolog Forum

July 6, 2007

28

With thanks to

Mark Musen and Natasha Noy

29

30

In biology, lots of ontology developers are almost hobbyists

• Nearly always, ontologies are created to address pressing practical needs

• Biologists ... may have little appreciation for metaphysics, principles of knowledge representation, or computational logic

• There simply aren’t enough good ontologists to go around

31

Issues in assuring ontology quality• Unlike the case with journal submissions, it makes no

sense for ontologies to be peer-reviewed by just a handful of experts

• Open, community-based review of ontologies may be haphazard and chaotic

• Top–down solutions may offer rigid review critieria at the expense of scalability

• There is a pressing need for empirical evaluation of methods for ontology evaluation

32

OBO Foundry must address lots of questions

• Can the top-down approach scale? How many ontologies can be managed by a small panel of curators?

• Who gets to reject an ontology on the basis of form or content? What is the appeals process? How do we know whom to believe?

• Who will curate the curators?

33

NCBO will offer

• Technology for uploading, browsing, and using biomedical ontologies

• Methods to make the online “publication” of ontologies more like that of journal articles

• Tools to enable the biomedical community to put ontologies to work on a daily basis

34

http://bioportal.bioontology.org

35Local Neighborhood view

Browsing/Visualizing Ontologies

36

Hierarchy-to-root view

38

Goals for BioPortal

• Web accessible repository of ontologies for the biomedical community– Archived locally– Anywhere in cyberspace

• Support for ontology– Peer review– Annotation (marginalia)– Versioning– Alignment– Search

39

Ontologies are not like journal articles

• It is difficult to judge methodological soundness simply by inspection

• We may wish to use an ontology even though some portions – Are not well designed– Make distinctions that are different from

those that we might want

40

Ontologies are not like journal articles

• The utility of ontologies– Depends on the task– May be highly subjective

• The expertise and biases of reviewers may vary widely with respect to different portions of an ontology

• Users should want the opinions of more than 2–3 hand-selected reviewers

• Peer review needs to scale to the entire user community

41

Community-Based Annotation

• Makes ontology evaluation a democratic process

• Assumes users’ applications of ontologies will lead to insights not achievable by inspection alone

• Assumes end-users will be motivated to comment on and engage in dialog about ontologies in the repository

Solution Snapshot

45

Open ratings for ontologies• Any user can

– rate an ontology– add a “marginal note”

• Ontology evaluation becomes a community-based initiative

• A web of trust can enable users to filter comments or ratings to avoid “noise”

Arguments in favor of the top-down approach in the scientific domain

• marginalia will contain a great deal of irrelevantalia

• scientists need ontologies, but are normally not experts in ontology; they are looking for authoritative guidelines

• the Foundry process is yielding guidelines on how to build ontologies compatible with those which already exist

46

Arguments against the top-down approach

• ontologies are not like journal articles, and it is difficult to judge methodological soundness simply by inspection.

• the evaluation process does not yield a quantifiable result.

– but scientific journals face exactly similar problems, yet peer review, there, works well

47

In defense of democratic rankings

• ranking by large numbers of users will tend to counteract such biases (but will the ranking service in fact attract users?)

• ranking by large numbers of users has a greater opportunity to scale up when ontologies proliferate

48

We have common goals

• Both approaches seek quality assurance to support ontology selection.

• Both approaches need to address the fact that the expertise and biases of reviewers may vary widely with respect to different ontologies or to different portions of an ontology.

49

One big difference

For Musen et al. there are no restrictions on entry

The bottom-up approach seeks community-based annotation of ontologies, with no difference being made between experts and non-experts

50

One big difference

In the OBO Foundry reviews are created precisely by the peers of the ontology authors themselves—by persons with established and recognized expertise and with a demonstrated willingness to invest due diligence in ontology development, use, and evaluation.

51

Both are needed

• in domains such as refrigerators

• but in science?

52