FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada

1

FME within NRCan

Marie-Eve Martin Jean-Marc Prévost

2

Agenda

!  Context !  System architecture !  Programming standards, Quality assurance,

testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration

3

Agenda

!  Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?

!  System architecture !  Programming standards, Quality assurance, testing methods, etc. !  Validations processes and dependencies !  FME developments at NRCan !  Safe Software collaboration

4

Natural Resources Canada Mission and Outcomes

5

NRC

an

Earth Science Sector

Geoma4cs Canada

CCMEO / CCCOT

Energy Sector

Canadian Forest Service

Direc4on & Coordina4on

Minerals & Metal Sector

Science & Policy Integra4on

Energy Technology & Programs Sector

Corporate Management & Services Sector

NRCan structure overview

CCMEO Canada Centre for Mapping and Earth Observa5on

CCCOT Centre Canadien de Cartographie et d’ Observa5on de la Terre

CCMEO total number of employees: 319 227 employees for Mapping 92 employees for Earth Observa5on

Sherbrooke

OGawa

6

Agenda



7

FME history at CCMEO

NTDB specs

vectoriza4on File management

CanVec, CanTopo, GeoBase, etc. era Database management

FGP 2015 2010 2005 2000 1995 1990 1985

Transla4ons -‐ Workbench

-‐ Data manipula4on & valida4on

Custom transformers

-‐ Programming standards

-‐ Tes4ng methodology

≈700

8

Agenda



9

Distribu?on Management

System architecture at CCMEO

Aquisi?on

Cond

i4on

ning

Partners

Data acquisition

Others

Data Access QA

10

What do we produce ?

!  Main products : !  Canvec !  Toporama !  CDED !  200 collections of various nature

!  Statistics for last year : !  8 500 000 downloads !  3 000 mail and phone requests

11

Where to find our products ?

!  Data availability : !  Geogratis : geogratis.gc.ca !  Geobase : www.geobase.ca ! WMS services

!  Licence : Open Government Licence

12

Data extraction tool http://geogratis.gc.ca/site/eng/extraction

13

Agenda



14

15

Agenda



16

System architecture

FME, Python, .EXE, SQL, etc.

Windows servers Produc4on: 64 processes (32bits) Distribu4on: 56 processes (64bits)

Oracle 11g, PostgreSQL 9.2 ArcSDE 10.1, PostGIS 2.0

400 TB

Instruc?on files

17

Agenda



18

Worbenches – Last saved build

Programs s4ll in opera4on !

19

Taking control

!  2007 problematic: !  Mix of FME versions on computers !  Everyone wanted the lasted build !  Increasing number of workbenches !  Workbenches were transferred to support team

!  Solution : !  FME upgrades management !  Programming standards

20

FME upgrades management

!  Upgrade rules: !  Everyone use one common build !  Upgrade only to official builds !  At least one upgrade per year

(idealy no more than two)

21

Programming standards

22

Programming standards

!  Documentation (.docx and annotations) !  Parameter naming standards !  Known bad practices review !  Unit tests !  Peer review

23

Programming standards Examples of bad practices

!  Missing Rounders !  Missing Terminators

24

Programming standards Unit tests

!  Each workbench is managed in a source code control environment

!  Each workbench is saved with the bat file, sources and expected results files necessary to test it

!  Batch testing of all workbenches before every upgrade.

25

Programming standards Unit tests

!  Fail rate is used to plan upgrades

Source Destination Fail rate

2013 SP1 2013 SP4 2%

2013 SP4 2014 SP1 10%

26

Programming standards Peer review

!  FME is easy to learn without taking classes !  FME always offer various way of doing

something

!  Peer review allows: !  Improved shared knowledge !  Programming uniformity !  Identifying repetitive patterns

27

Agenda



28

Validation processes

!  At CCMEO, FME is used for : !  Feature translation !  Feature creation !  Feature validation

Catalog

29

Distribu?on Management

Validation processes

Aquisi?on

Cond

i4on

ning

Partners

Data acquisition

Others

Data Access QA

30

Validation Catalog

!  ISO-19110 compliant

XML HTML Database

Users

31

Catalog based validations

!  Spatial relations validation !  Domain attribute validation !  Proximity validation !  Minimal dimension validation !  Segmentation validation !  Data clipping validation

32

Catalog based validations Example

Spatial relations validation !  Based on Egenhofer-Clementini DE-9IM masks

33

Catalog based validations Examples

!  Spatial relations

!  Attribute value

34

Catalog validation within FME

ISO 19110 XML

List friendly format

Generic reader

Generic writer

Valida4on Error

forma4ng Valida4on rules

extrac4on Valida4on

Error forma4ng

Valida4on rules extrac4on

One custo

m transfor

mer

per valida

5on conc

ept!

35

Catalog validation within FME

!  Real program example:

36

Agenda



37

Utility customs - Examples

!  TerminatorRecorder !  Record all the errors to a .ffs file before causing the workbench to fail

!  TerminatorMinMax !  Cause the workbench to fail if

the number of feature is not as expected

38

!  Matcher_CCMEO ! Match features if

start/end points are within tolerance

!  SnapToNeatline !  Snap features within tolerance without

collapsing


39


!  SchemaMapper_CCMEO !  Similar to SchemaMapper but : !  Allow mapping with a variable number of

conditions !  Conditions may be ranges of values !  Conditions may use any operator (>, <, != , etc) !  Conditions are validated and produce report if

badly formatted

TYPE=[11:13];PRECISION<=20;NAME=Lake

40

Validation customs Minimal dimension

Voronoi

Side lines

Centerline

Area Width Length

41

Validation customs Proximity

What are we looking for?

Valida4on Process

Op4misa4on Process

42

Agenda



43

Safe Software collaboration

!  Motivations for collaboration !  Simplifies our workbenches !  Faster processing ! Maintenance under Safe’s responsibility

44

Safe Software collaboration Generalization

Hydro network

AngularityCalculator

SherbendGeneralizer

StreamOrderCalculator

StreamPriorityCalculator

NetworkFlowOrientor

BoundingBoxAccumulator

45

Safe Software collaboration

VoronoiDiagrammer

46

QUESTIONS ?!?

[email protected] [email protected]

47

Amalgamator

48

ConvexityFilter

49

Generalizer Wang The Wang algorithm will itera5vely combine, eliminate and exaggerate bends un5l the input line feature has no bend that is smaller than the given tolerance value.

McMaster / McMaster Weighted Distance The McMaster algorithm calculates a new loca5on for each point by first taking the average value of the x and y coordinates of the point and a number of neighboring points. It then slides the averaged point towards the original point according to a specified displacement value. The overall effect is that each point will be pulled towards its neighboring points.

NURBfit (B-‐Spline) The NURBfit algorithm will fit lines using B-‐Spline curves of given polynomial degree. The resul5ng lines will follow these curves with given segment length. The higher the degree, the smoother the line. An example of usage is smoothing contour lines in order to remove sparks and simulate the work of a cartographic cra[sman.

Inflec?on Points The Inflec5on algorithm will calculate the loca5on of the inflec5on points along a line and return the list of these points. Inflec5on points are measures of the sinuosity of a line.

50

SherbendGeneralizer SherbendGeneralizer

51

NetworkFlowOrientor NetworkFlowOrientor

52



53

Densifier

54

Chopper

55

Chopper use case scenario

chopper

dissolve

chopper chopper

dissolve

dissolve

Data extraction

Group by metadata No group by

Date post:	11-Jun-2015
Category:	Technology
Upload:	safe-software
View:	4,737 times
Download:	2 times