Beispielbild The BioCASe Technology Jörg Holetschek Botanic Garden & Botanical Museum Berlin-Dahlem...

transcript

BeispielbildThe BioCASe

Technology

Jörg HoletschekBotanic Garden & Botanical Museum Berlin-DahlemDept. of Biodiversity Informatics and LaboratoriesKönigin-Luise-Straße 6-814195 Berlin

BioCASe Workshop, Melbourne, Feb 4-5th 2010

2BioCASe Workshop, Melbourne, Feb 4-5th 2010

Agenda

1. The BioCASe Architecture: An Overview

2. The BioCASe Provider SoftwareFeature, Requirements, Installation, Configuration

3. The ABCD and HISPID data standardsIntentions, Structure, Elements, Use

4. Preparing the database for BioCASe/ABCD

5. Setting Up DatasourcesDatabase connection, Table Setup, Mapping Process;Testing, Data Backups

6. Other Issues

Workshop Wiki: http://hiscom.chah.org.au/wiki/BioCASe_Workshop

Beispielbild

1. BioCASe Technology:

Motivation, Idea and Architecture

Herbaria, Drawings

Preserved Specimens

Living Collections

Culture collections

Primary Biodiversity Data

Biological Collection Access Service

Documentation of the occurrence of one species

at a given location at a certain point in time

= Primary Biodiversity Data Record

Data sources worldwide

- Index Herbariorum: 3,293 herbaria, 400 million herbarium sheets- 50-100,000 natural history collections, 1.5-2 billion specimens- With observations added, occurrence records 3+ billion (10b?)

Over 75% of biodiversity information are stored in developed countries.

Est. 75% of all species are found in the developing world.

Source: BARTHLOTT et al. 1999

Accessibility

Stage 0: Only in real world (paper catalogues, just stacks)Only meta information available on the web

Stage 1: Stage 2: Online catalogue Digitalization of specimen

Biodiversity Data

Stage 3: Networking the databases

Architectural Overview

2. Wrapper Software

1. Protocols/Data Standards

Data Quality CheckerDataMining

3. Applications

Data Portal

BioCASe Design Principles

No central database Data remain in the existing DB systems Data Provider gets full credit Full control over published data by collection holder

Partial publication possible

Collection holder can withhold information from publication (e.g., locality data for endangered species) or exclude records (e.g. until research results are published)

Wrapper principle Data remain in original collection management system No changes in workflow for curator/local users

2: The BioCASe ProviderSoftware

Wrapper: BioCASeProvider Software

Protocols/Data Standards

Applications

Data Portal

Software package that „wraps“ around the collection database Equips it with a BioCASe protocol compliant interface

1. Accepts requests from the network

3. Transforms results into ABCD documents and sends them back

BioCASe Provider Software (Wrapper)

Marmotamarmota?

2. Tanslates queries to the collection database

SELECT *FROM specimenWHERE ScientificName LIKE “Marmota marmota%“

BioCASe Provider Software (Wrapper)

Compatible with several protocols (BioCASe, DiGIR) and data schemas (ABCD, ABCD-EFG, ABCD-DNA, DarwinCore)

Works with all SQL-compliant databases (Access, mySQL, Postgres, SQL Server, Oracle, ...)

Currently 84 production installations serving 1.315 collections

Platform independent Support

available!

BioCASe Providers Worldwide

84 production installationsserving 1.315 collections

Requirements

- SQL compliant database (with existing Python connectivity module)

- Webserver (preferrably Apache), allowing the execution of Python scripts

- Possibility to install additional Python packages

1. Installing Apache

2. Installing Python

3. Downloading BPS (from repository/archive)

4. Installing BPS

5. Creating the link Apache/BPS

6. Testing BPS, Setup of additional packages

7. Changing directory permissions

1. Installing Apache

http://httpd.apache.org/download

2. Installing Python

http://www.python.org/download/

3. Downloading BPS

Archive: http://www.biocase.org/products/provider_software/

Subversion repository

Trunk version: http://ww2.biocase.org/svn/bps2/trunk

Defined version: http://ww2.biocase.org/svn/bps2/tags/release_2.5.2

Linux:

svn co <url> <path>

Windows: Tortoise client

4. Installing the BPS

Setup.py

No files copies,

only adapted!

5. Linking BPS with Apache

http.conf

6. Testing BPS, Installing Additional Packages

http://localhost/biocase Utilities Library Test

6a: mysqldb

http://sourceforge.net/projects/mysql-python/

7. Write permissions

... /bps2/configuration

Changing the Password

... /bps/configuration.ini

3: ABCD, HISPID

Protocols/Data Standards

Wrapper Software

Applications

Data Portal

ABCD Data Schema

Access to Biological Collection Data:

Data schema for all types of primary biodiversity data (living/preserved/observational, botanical/zoological/bacterial/viral, marine/terrestrial)

XML (eXtensible Markup Language) based can be consumed by humans and machines

Highly complex, hierarchical, currently 1,055 data elements almost every data item will fit in

Extendable (plug-in slot for additional information)

standard (currently version 2.06)

ABCD: Structure

Namespace: http://www.tdwg.org/schemas/abcd/2.06

ABCD Metadata: Technical/Content Contact

ABCD Metadata: Intellectual Property Rights

ABCD Metadata: Representation, Owner, ...

ABCD: Triple ID, Record Basis

ABCD: Identification (multiple)

ABCD: Gathering Event

ABCD: Multimedia

ABCD: Unit Associations

ABCD: Specialised Portions

Specimen Unit: Acquisition, Accession, Peparation, Duplicate Distribution, Type Status

Herbarium Unit: Loan Information

Botanical Garden Unit: Location in Garden, Hardiness, Lineage, Cultivation, Planting Date

Other Specialised Subtrees forObservationsCulture CollectionsMycological UnitsZoological UnitsPaleontological UnitsPlant Genetic Resources

ABCD: UnitExtension

Own Namespace for Extension http://www.chah.org.au/schemas/hispid/5

Other Extensions: Extension for Geoscienes (ABCD-EFG) DNA Bank Network (ABCD-DNA)

HISPID

HISPID Gathering-Coordinates DMS-PersonCollector-Substrate/ParentRock-SoilType, Vegetation

HISPID Unit-LifeForm, Phenology-NonComputerisedDataFlag-DonorTyp, ProvenanceType

HISPID Identification

-HigherTaxon: Addition ranks

BioCASe Protocol

Biological Collection Access Service Protocol:

Manages data exchange between data providers (collections) and applications (data portals)

Vehicle for transporting requests: data portal collection and responses (ABCD documents): collection database data portal

XML based

BioCASe Protocol: Capabilities request

BioCASe Protocol: Inventory Request

BioCASe Protocol: Search Request

Beispielbild

4. Preparing the database for BioCASe

4. Reasons for not publishing the live DB

1. Publishing the live DB is not desired creating snapshots for publication

2. DBMS not accessible for the BPS export into another DBMS

3. Performance considerations (too highly normalized) partial, controlled denormalization

4. Repeatable elements kept in columns, not in separate rows Moving repeatable elements to separate records

Each repeatable elements needs its own primary key!

Repeatable elements kept in columns

specimen_id ... class order family

3476 ... Conjugatophyceae Desmidiales Desmidiaceae

3477 ... Conjugatophyceae Desmidiales Desmidiaceae

3478 ... Conjugatophyceae Desmidiales Closteriaceae

specimen_id ...

3476 ...

3477 ...

3478 ...

sp_id ht_entry ht_rank ht_name

3476 456765 class Conjugatophyceae

3476 456766 order Desmidiales

3476 456767 family Desmidiaceae

3477 456770 family Desmidiaceae

3478 456773 family Closteriaceae

CREATE VIEW [dbo].[vwHigherTaxa]

SELECT 'k_' + [EDIT_ATBI_RecordID] AS id, [EDIT_ATBI_RecordID] AS unit_id, [kingdom] AS name, 'kingdom' AS rankFROM unit_dataWHERE [kingdom] IS NOT NULL

SELECT 'p_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [phylum], 'phylum'FROM unit_dataWHERE [phylum] IS NOT NULL

SELECT 'c_' + [EDIT_ATBI_RecordID], [EDIT_ATBI_RecordID], [class], 'class'FROM unit_dataWHERE [class] IS NOT NULL

Example View

Commonly used repeatable elements

- Identification- HigherTaxon- GatheringSite/NamedArea- Metadata/Scope/GeoecologicalTerms- Metadata/Scope/TaxonomicTerms- MultimediaObjects- MeasurementsOrFacts- ...

Controlled Denormalization

insert into [dbo].[abcd_Object]

SELECT dbo.CollectionObject.CollectionObjectID, ISNULL(dbo.CatalogSeries.SeriesName, '') + '-' + ISNULL(CAST(dbo.CollectionObjectCatalog.SubNumber AS nvarchar(20)), '') + '-' + ISNULL(CAST(dbo.CollectionObjectCatalog.CatalogNumber AS nvarchar(20)), ''), dbo.f_getParentID(dbo.CollectionObject.CollectionObjectID), dbo.f_getCollectingEventID(dbo.CollectionObject.CollectionObjectID), dbo.f_getFieldNumber(dbo.CollectionObject.CollectionObjectID), cast(dbo.CollectionObjectCatalog.CatalogNumber as int), dbo.CollectionObject.PreparationMethod, case when Sex = '<No Data>' then NULL else Sex end, case when Stage = '<No Data>' then NULL else Stage end, case when dbo.CollectionObject.Text1 is null then '' else 'Barcode: ' + dbo.CollectionObject.Text1 + '; ' end + case when dbo.Accession.Number is null then '' else 'Specimen Location: ' + dbo.Accession.Number end + case when DerivedFrom.Remarks is null then '' else ' <br> ' + cast(DerivedFrom.Remarks as nvarchar(2000)) end

FROM dbo.BiologicalObjectAttributes RIGHT OUTER JOIN dbo.CollectionObject ON dbo.BiologicalObjectAttributes.BiologicalObjectAttributesID = dbo.f_getParentID(dbo.CollectionObject.CollectionObjectID)

LEFT OUTER JOIN dbo.CollectionObjectCatalog LEFT OUTER JOIN dbo.CatalogSeries ON dbo.CollectionObjectCatalog.CatalogSeriesID = dbo.CatalogSeries.CatalogSeriesID ON dbo.CollectionObject.CollectionObjectID = dbo.CollectionObjectCatalog.CollectionObjectCatalogID

LEFT JOIN dbo.Accession on Accession.AccessionID = CollectionObjectCatalog.AccessionID

LEFT JOIN dbo.CollectionObject AS DerivedFrom ON CollectionObject.DerivedFromID = DerivedFrom.collectionObjectID

WHERE (dbo.f_hasChildObjects(dbo.CollectionObject.CollectionObjectID) = 0) AND ...

How Do I See Someting is Wrong?

Errors in ABCD documents:-Several datasets (one for each unit)-Several units for one specimen record

Reasons:- Repeatable elements not in separate tables (no separate PK several units will be created)- Several records in DB for non-repeatable elements (several ABCD objects are necessary to create a valid document)

Beispielbild

5. Setting Up a BioCASe Data Source:Database connection, Table Setup, Schema Mapping

BPS Datasource

URL for a BioCASe protocol compliant webservice:http://ww3.bgbm.org/biocase/pywrapper.cgi?dsa=AlgenEngels

<?xml version='1.0' encoding='UTF-8'?><request xmlns='http://www.biocase.org/schemas/protocol/1.3'> <header> <type>search</type> </header> <search> <requestFormat>http://www.tdwg.org/schemas/abcd/2.06</requestFormat> <responseFormat start='0' limit='10'> http://www.tdwg.org/schemas/abcd/2.06</responseFormat> <filter> <like path='/DataSets/DataSet/Units/Unit/Identifications/Identification/ Result/TaxonIdentified/ScientificName/FullScientificNameString'>A*</like> </filter> <count>false</count> </search></request>

BPS QueryTools

Tool for sending Scan, Search and Capabilities Requests to a datasource

Choose datasource „Debug“

Steps for Setting Up a Datasource

1. Create a new Datasource

2. Configure Datasource:1. Database Connection

2. Table Setup

3. Create new empty Mapping

4. Edit Mapping:

1. Choose root table

2. Edit mandatory ABCD elements (red)

3. Save Configration, test datasource (QueryTools)

4. Add additional ABCD elements, occasional testing

3. Test Datasource

Datasource Loglevel

The lower the loglevel, the more information is logged.(10=info, 20=debug, 30=warning, 40=error)

How The BPS performs requests

1. Get a ID list of records matching the filter

2. Loading all details for the matching IDs Joining of ALL tables, beginning with the root table (table with UnitID, one record per Unit)

Datasources folder

... /bps/configuration/datasources/<dsname>

querytool_prefs.xmlJust what its name says.

xxxx.pickTemporary files; should be deleted if BPS behaves strangely.

cmf_xxxxxx.xmlConcept mapping; one for each supported schema.

provider_setup_file.xmlDatabase conncetion, table setup, supported schemas.

Regular backup of configuration folder is highly recommended!

Metadata tables

If metadata differ for each or some of the records: several records in metadata table, linked to unit by foreign key

If metadata is unique for all records possible to hold data in one record no reference key is needed static table

Applications

2. Wrapper Software

1. Protocols/Data Standards

3. Applications

Data Portal

Local QueryTool

Distibuted Search vs. Harvesting/Caching

GeoCASe Distributed Search: http://search.biocase.org/geocase

GBIF Registration

GBIF Data Portal

BioCASe European data portal

EDIT Specimen Explorer

Data Mining: Itineraries Project

Goal: Detect itinerary patterns in geo-referenced primary data presumably collected during a collecting event.

1st step:

Try to validate itineraries from well-documented expeditions (literature) against geo-referenced primary biodiversity records with dates/collecting information

2nd step:

Try to find itineraries for collecting events with missing expedition diaries

Data Mining: Ecological Niche Modelling

Jörg Holetschek

Botanischer Garten & Botanisches MuseumAbteilung Biodiversitätsinformatik & Labors

Königin-Luise-Straße 6-814195 Berlin-Dahlem

j.holetschek@bgbm.orgTel. +49 30 838 50150

0448 831 980

www.bgbm.org/biodivinf

www.biocase.orgsearch.biocase.orgsearch.biocase.de

http://hiscom.chah.org.au/wiki/BioCASe_Workshop

Beispielbild The BioCASe Technology Jörg Holetschek Botanic Garden & Botanical Museum Berlin-Dahlem...

Documents