DataReceiver and SchemaMapperPeter Laulund - Agency for Data Supply and Efficiency

AbstractAt SDFE we are working on a project, where we have to import up to100 feature types in a number of formats into Oracle, using FME andFMEServer. In order to manage this, we have created one workspacethat handle all datasets and formats, a key to this is theSchemaMapper. The SchemaMapper is a transformer with a badreputation, but as I will show it a flexible and dynamic transformer whenyou use it in combination with a database. I will also demonstrate howwe can use an expression as part of the schema mapping. In mypresentation I will discuss this project with special focus on theSchemaMapper.

About me

• Working with FME since 1997• Working with FMEServer since 2001• Certified professional• Works at SDFE (KMS, GST)

Favorite transformers

SDFE and FME

• We use FME and FMEServer• Major projects

• Kortforsyningen.download.dk• INSPIRE• Transportbåndet

• Datamodtagelsen / DataReceiver

DataReceiver

DataReceiver

• Up to +100 feature types• Any input format -> Oracle spatial

• Mapinfo, Shape, FFS, GML, CSV, XLS, ….

• Any coordinate system -> EPSG:25832• Action: Total, Replace, Synchronize and

Delta

The challenge

• One workspace that handles all featuretypes

• Documenting the translation

PL/SQL

Infrastructure

JMSPollingService

File area

FMEServerSFTP

Log

Log

Staging Test Prod

admin.web

PL/SQL

main.fmw

FMEServer

Log

Stagingload2oracle.fmwFile area

successerrormail

jms / json

FMEServer

main.fmw

{ "jms_publisher_content" : {"dm_jobid" : 10116,"prs_id" : 1,"filepath" : "c00016/00000000013","seq_nbr" : "13","cha_id" : 16,"prior_seq" : 12,"prs_name" : "load"

},"jms_publisher_type" : "text","fns_type" : "jms_publisher"}

load2oracle.fmw

load2oracle.fmw - Reader

SchemaMapper

Actually, I think the SchemaMappercould use some love, it feels a bitclunky compared to other and morerecent additions in FME

david_r on FME Community

The SchemaMapper transformercan be very handy but it definitelyhas it's limitations and it feels likeit's getting a bit long in the tooth

nic_ran on FME Community

SchemaMapper functionality

• Filter incoming features• Rename attribute• Add an attribute with default value• Default can be an expression• Attribute lookup with filter

SchemaMapper configurationFilter Value Old Attr New Attr Default

fme_feature_type tower name MASTENUMMER

fme_feature_type tower sdfe_timestamp REGISTRERINGFRA

fme_feature_type tower profile MASTETYPE unknown

fme_feature_type tower towerheight MASTEHOEJDE -1

fme_feature_type tower ID_LOKALID @UUID()

fme_feature_type tower ID_NAMESPACE data.gov.dk\Højspændingsnet\Elmast

fme_feature_type tower sdfe_feature_type ELMAST

fme_feature_type tower sdfe_geom_coordsys EPSG:25832

fme_feature_type tower SENESTEREVISION @DateTimeParse(@Value(extpub_dato)),ISO)

fme_feature_type tower STATUS @LowerCase(@Value(tatus))

Attribut Expresion ValueAREA @Area() 10000

LENGTH @Length() 400

UUID @UUid() d5c31968-a5e0-4b90-8967-8b716df34d2e

EVALUATE @Evaluate(@Value(AREA) * @Value(LENGTH)) 4000000

CONCATENATE @Concatenate(area is, @Value(AREA), \,and length is,@Value(LENGTH))

area is 10000 ,and length is 400

A @Value(A) Hello world

TCL @Tcl2(myProc) Hello Peter

COORDSYS @CoordSys() EPSG:25832

GEOM_IS_VALID @OGCGeometry(is_valid) True

NOW @DateTimeNow() 20190227163039.4223444

UPPER @UpperCase(@Value(A)) HELLO WORLD

SchemaMapper and functions

String and date functions

• @LowerCase(), @DateTimeNow()• @EvaluateExpression(<expression>)• Expression must be WWJD encoded• Use a PL/SQL function in a view to do

the encoding

WWJD encoding in OracleCREATE FUNCTION fme_encode( i_s in varchar2 )RETURN varchar2 IS

s varchar2(4000) ;BEGIN

s := replace(i_s, '<', ‘<lt>' ) ;s := replace(s, '>', '<gt>' ) ;………s := replace(s, q'{'}', '<apos>' ) ;s := '@EvaluateExpression(unused1,' || s || ',unused2)' ;return s;

END;

@EvaluateExpression(unused1,<at>LowerCase<openparen><at>Value<openparen>sportype<closeparen><closeparen>

, unused2)

DECODE(ENCODE, 1, DM_ADM.FME_ENCODE(DEFAULT_VAL), DEFAULT_VAL) DEFAULT_VAL

FUNCTION fme_encode( i_s in varchar2 ) RETURN varchar2IS

s varchar2(4000) ;BEGIN

s := replace(i_s, '<', ':lt:' ) ;s := replace(s, '>', '<gt>' ) ;s := replace(s, ':lt:', '<lt>' ) ;s := replace(s, '(', '<openparen>' ) ;s := replace(s, ')', '<closeparen>' ) ;s := replace(s, '{', '<opencurly>' ) ;s := replace(s, '}', '<closecurly>' ) ;s := replace(s, '[', '<openbracket>' ) ;s := replace(s, ']', '<closebracket>' ) ;s := replace(s, '$', '<dollar>' ) ;s := replace(s, '@', '<at>' ) ;s := replace(s, ' ', '<space>' ) ;s := replace(s, ';', '<semicolon>' ) ;s := replace(s, ',', '<comma>' ) ;s := replace(s, '"', '<quote>' ) ;s := replace(s, '/', '<solidus>' ) ;s := replace(s, '\', '<backslash>' ) ;s := replace(s, q'{'}', '<apos>' ) ;s := '@EvaluateExpression(unused1,' || s || ',unused2)' ;return s;

END;

SchemaMapper

• Metadata in a database• Use Where when reading• Map metadata with a view• Expressions as default value• Encode functions in view

Summary

• FME is an key part of our infrastructure• One FME project• Metadata document the mapping• Functions in SchemaMapper• I love SchemaMapper:-}

QuestionsPeter Laulund – pelau@sdfe.dk