Constructing a Federated Data Distribution Service for Universities

transcript

CONNECT. TRANSFORM. AUTOMATE.

Jakob Lagerstedt, Mats HögströmConstructing a Federated Data Distribution Service for Universities

Overview

Background

Previous Attempts

A modern solution

Some tricks learned

Results

Background

We are from the

Swedish University of Agricultural Sciences

Background

Data and researchers in Sweden

Background

Not free

Image courtesy of KJGarbutt (CC BY 2.0)

Background

Strange formats

Image courtesy of Toronto Public Library Special Collections (CC BY 2.0)

Previous attempts

Distribution was difficultRequired authorization and data preparation

Researchers would share pieces of data on USB drives (sneakernets)

Image from http://japanese.engadget.com/2005/11/12/hdd-circa-1975/

A modern solution

Geographic Extraction Tool

A modern solution

Tricks of the trade ()

Do not panic

All the code is in the handouts if you are interested!

Image courtesy of Nico Kaiser (CC BY 2.0)

FME server Rest API

Token service /fmetoken/service/generate

HTTP get /fmerest/repositories/" + your_fme_repo_name + ".json

Tricks of the trade (Servlet)

Image courtesy of abraham.williams (CC BY 2.0)

Tricks of the trade (Shutdown Python script)

To enable unsupported functions

Zip & Email & Remove files that have been downloaded, Anything you want!

Tricks of the trade (External script example 1 handout)

Use an external script to enable faster debugging

Monkey patch sys or import something from path (yes we actually use the linux version)

Tricks of the trade (SpatiaLite example 2 handout)

Wonderful format for mixed environments

We have Qgis, Arcmap, FME, they all read it!

Image courtesy of juicyverve (CC BY 2.0)

Tricks of the trade (SpatiaLite)

However, FME does not read as efficiently as other formats

does not set spatial index

Can fix

Tricks of the trade (Google Fusion Tables)

Fusion tables are nice for reporting

But often writes fail (we do not know why, reported the issue)

Tricks of the trade (KML network link)

KMLs are nice for reporting

Results

23 Universities

100K distributed data sets ( 400 * total area of country)

6000 Unique users

Image courtesy of saamiblog (CC BY 2.0)

Thank You!

Questions? Handout link

http://pluto.slu.se/drop/public.php?service=files&t=2e0c13550e0c71e963e6f6bb2c52569c

For more information: jakob.lagerstedt | mats.hogstrom @slu.se Swedish University of Agricultural Sciences

Constructing a Federated Data Distribution Service for Universities

Technology