Back to FME School - Day 2: Your Data and FME

transcript

Back to FME School3-Day webinar series

Course ScheduleTuesday:

Your Data and FME

Wednesday: Automating Everything

Thursday:Expanding Frontiers

FME Workbench Performance Tips & Tricks

A Database for Every Occasion

Working with Attributes in FME

Bulk data processing

Ultimate Real-Time: Monitor Anything, Update Anything

FME in the Enterprise

Remote sensing providers

Web and business data into gold

New + Upcoming in FME Cloud

40-minute talks

Let’s begin today’s lesson...

Potent Potions for Batch Data

Processing

The Heavy Lifters: Two key Transformers for Bulk Data Processing

Meet the

FeatureReader & FeatureWriter

FeatureReader (since FME 2011)

Solves problems such as:

1. Read data within a selected boundary

2. Read data within a user drawn area3. Process a directory of files

4. Read in data FME has written in the same workspace

FeatureReader

FeatureWriter (since FME 2016)

Do things AFTER the data is written in FME Workbench.

Less need for Shutdown Python or TCL

● copy data after writing● FTP results● Email an attachment● Load into Dropbox or S3● Insert data into HTML file● Database validation● FME Server Notification● 3rd Party Integration● Can Write in Parallel!

FeatureReader and WriterIn the Real World

Web Client with FME

FeatureReader & WriterIn the Real World

FeatureWriter Demo

Now Let’sBatch...

250,000 CAD files & rasterson mobile devices

Tip: Know Your Potions and Choose Wisely

Today’s Potions1. Wildcards

2. Batch Deploy

3. Parent/child Workspaces

4. Parent/child Server Workspaces

Potion 1: One Wild<card>

Dataset

Multi-Dataset Picker

ShapefileMapInfo

Most rasters

DWGDGNSQLite

Dataset Wildcards

Extended glob syntax:

Symbol Matches

? Any single character

* Any sequence of zero or more characters

[chars] Any single character in chars.

[a-d] Any character between a and d inclusive

{a,b,...} Any of the sub-patterns a, b

/**/ 0 or more subdirectories

Time to brewPotion 1

Wildcard Bulk Data Processing Enticements

✓ Simple to set up

✓ Can transform across file boundaries

- Needs memory & time

Wildcard Bulk Data Processing

Pitfalls

x Recovery from data errors difficult

x Feature Type vs File vs Format Issues

x No granular logx No ability to

parallelize

Potion 2: Batch Deploy

Batch Deploy Script Writer

Batch Deploy Enticements

✓ Simple to set up✓ Runs quickly✓ Can script via

command line✓ Run on demand

Batch Deploy Pitfalls

x Recovery from data errors difficult

x No granular logx Destination dataset

naming can be tricky

Potion 3: Parent/ChildWorkspaces

Parent/Child Workspace Ingredients

• Parent Workspace:– PathReader– WorkspaceRunner

• Child Workspace:– FeatureWriter

Parent Ingredients

Child Ingredients

Parent/Child Workspace Enticements

✓ Separate transformation from workflow

✓ Generate audit logs✓ All authored within

Workbench

Parent/ChildWorkspace

Pitfalls

x Not all writers can be used concurrently

x Slow to run each child workspace separately

x Recovery from data errors not easy if concurrent runs used

Potion 4: Parent/Child Server

Workspaces

Parent/Child Server

Workspace Ingredients

• Parent Workspace:– PathReader– FMEServerJobSubmitter– FMEServerJobWaiter

• Child Workspace:– FeatureWriter

Parent Ingredients

Child Ingredients

Parent/Child Server

Workspace Enticements

✓ Separate transformation from workflow

✓ Generate audit logs✓ All authored within

Workbench✓ Make full use of

parallelism = FAST

Parent/ChildServer

Workspace Pitfalls

x Not all writers can be used concurrently

x Data needs to be accessible to Server Engines- Consider using Server Data Resources

x Craft your reload/audit plan

Summary● Many ways to handle bulk data moves

● Choose your potion wisely - each has pluses and minuses

● FeatureReader/FeatureWriter increase flexibility

● FME Server is the most robust automation choice

Questions?

Batch processing tutorial: fme.ly/b59

Ultimate Real-Time Monitor Anything, Update Anything

FME Server Notification Service lets you act on events as they happen, and send information as it becomes available.

Publications and Subscriptions let you publish to and monitor different systems and data.

NotificationsWhat they are for

What they are not for

✓ A brief message, usually to trigger an action.

What they are not forx Transmitting large amounts

of spatial data.

✓ Triggering an FME Server response to an event that happened outside of FME.

of spatial data.x Triggering an FME Server

response to a continuous series of messages (many per second).

✓ Triggering an FME Server response to an event that happened outside of FME.

✓ Sending a message about something that happened in FME.

of spatial data.x Triggering an FME Server

response to a continuous series of messages (many per second).

x Sending more than one message per second about what’s happening in FME.

FME Server Notification Service

See the list of what built-in publications and subscribers can be: http://fme.ly/protocols

Workflows for“Ultimate Real-Time”

Poll it Push it Update it

Poll it“Are we there yet?”

Polling Workflow1. Change detection

○ GeoRSS reader○ Caching○ Timestamps

2. Run the workspace on a scheduleTimestamp

Scheduling (FME Server)

GeoRSS Reader

• Contains change detection logic

• Option to only read new features

Caching

• Whole datasets or timestamps

• ChangeDetector

• SQL - match on ID

ChangeDetector Example

SQL to find different features:@Value(uuid) = "uuid" AND @Value(aqi) != "aqi"

SQL Example

Timestamps

● Cache the timestamp

● Last updated attribute

● Filter features newer than timestamp

● Add a timestamp attributes to your dataset

Reads last cached time and replaces with current time

Checks against timestamp and filters out old features

Pros and ConsCaching whole datasets

✓ Easy to set up and detect changes.x Need to store entire

dataset - processing time?

Caching timestamps

✓ No storage needed.x Harder to set up -

variables, datetimes.

Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?

Story: Real-Time Traffic

✓ Poll Waze feed

✓ Parse XML

✓ Filter

✓ Transform

✓ Validate

✓ Email notifications

Push It

Pushing Methods• Database triggers• Webhooks

In FME Server:• Direct URL• REST API

Database Triggers

● Databases can trigger events.

● Functions with HTTP requests can push data to FME Server.

Database Triggers

Webhooks

● i.e. Give systems/services a URL that will respond.

● Send the notification to FME Server.

● Queue the received notifications and process the requests on a separate thread.

Pros and ConsDatabase Triggers

✓ Pushes right to FME Server topic.

✓ Can push lots of changes to a holding table.

✓ Perfect for real-time database changes.x Database permissions.

Webhooks

✓ No need to waste effort polling.

✓ Truly real time.x Complex to set up.x Not all systems provide

or accept webhooks.

Direct URLSystems can use HTTP to push data to topics or trigger workspaces to run.

Topic POST URL

REST APIIn-depth interaction

with FME Server

● Send/receive notifications● Run jobs● Manage connections● Manage users● Licensing● Backups● Scheduling● Etc!

To Poll or Push?

Poll vs. Push Considerations

● Simplicity vs. “real-time”ness● Can the system push to FME Server?● Are jobs too long for polling?● API restrictions for polling?

Tip: Use FME Server for internal systems and FME Cloud for external systems.

Update It

Updating Methods

• FME Writerso Bulk

drop/load/overwriteo Update option

• SQLExecuter• HTTPCaller

FME Writers: UPDATE operation

INSERT, UPDATE or DELETE in the middle of a workflow.

POST data or changes to systems.

Summary

• Polling workflow:o Change detection + scheduling

• Pushing options:o Database triggers, webhookso FME Server: URL or REST API

• Updating:o Operations in writer

parameterso SQLExecutoro HTTPCaller

Questions?

Database triggers tutorial:fme.ly/dbtriggers

Webhooks etc:fme.ly/web

REST API documentation:fme.ly/restv3

FME Server: Charms for the

Enterprise

Enterprise-Level Charms

★ Security - a new level with new charms

★ Deployment - move solutions easily

★ Automate the enterprise

★ Jobs Control

Security Charm #1Active Directory

✓ Any number of Active Directory domains✓ Any number of users per domain✓ A hybrid model - mix server and AD users

Active Directory

Functionality based on feedback and experience from users! Thank you!

Active Directory Support

Import Active Directory

Hybrid security model

Over to FME

Security Charm #2

Sharing and user control

✓ Users can share with other users or groups.

✓ Sharing defined for everything except “shared resources”.

✓ Users are isolated from each other.

✓ No load on Administrator.

Familiar Sharing Model (Google, Dropbox, etc.)

Modern Sharing Model

User-Controlled Security

✓ Users have control of all FME Server components

✓ Similar to state-of-the-art services like Google Drive.

Security Charm #3Connections & 3rd-Party Applications

Connection Based 3rd Party Integration

Why use connections?

● Encapsulate user credentials in a secure manner.

● Define for Databases and Web Applications

● Share across multiple workspaces.

● Share without revealing credentials.

Connection Based 3rd Party Integration

Many web connections.● Amazon Web Services● OAuth 2.0 (tens of

thousands of services)● Token● HTTP Authentication

All FME-supported databases.

Database Connections

Web ConnectionsMany web connections are shipped with FME

Web Connections

More are being added toFME Hub all the time

Security Charm #4We secure your infrastructure too.

On Premises vs FME Cloud

On-premise:

● you secure your infrastructure

● you configure FME Server security

FME Cloud:

● you configure FME Server security

FME Cloud SecurityWe manage data, application and transmission security.

● HTTPS only

● Annual security audit

● Automated security patching

● Leverage AWS Compliance - Industry leaders

● Control firewall rules (protocols & ports)

Deployment CharmEasily move solutions

Challenge: How can you move an FME Server

solution from one server to another?

Your FME Server solutions usually consist of many components.

Workspaces Subscriptions Schedules

Publications Resources Connections

Custom Formats Custom Transformers Topics

FME Server Projects

Portable Server SolutionsPlace your screenshot here

FME Server ProjectsPortable way to move, distribute, and share FME Server solutions.

Benefit #1 - Move a project from one machine to another.

Development StagingTest Production

Projects.

Benefit #2 - Backup (export) a project any time for archiving.

FME Server Projects

Projects.

Benefit #3 - Share your projects.

First - Export your project!

Projects.

Project Export

FME Server Projects

Projects.

Benefit #3 - Share your projects

Next - Send to those with whom you want to share it.

Projects.

Project Export

FME Server Projects

Projects.

Benefit #3 - Share your projects Finally - Receivers simply import the project.

Projects.

Project Export

FME Server Projects

Projects.

Benefit #4 - Publish Projects to FME Hub. (Coming Soon)

Share FME Server solutions with the FME community.

Projects.

Project Export

FME Server Projects

Automation Charm

EnterpriseAutomation

This charm is so powerful that we have another presentation

just on this.

Termination Charm

Goal: Ensure FME Engines don’t get stuck on a job.

Workflows are often dependent on 3rd Party applications/services.

An engine may hang or take too long waiting on responses.

For any job you now have a new setting (Schedules and run Workspace)

Max elapsed time in seconds job will run.If time expires job is killed!

Charm: Specify maximum time a job can run.

If job takes longer than allocated then the job is cancelled.

Summary - FME Server 2017 Charms✓ Enterprise Level Security: Enable fine-tuned control with

the new security model.

✓ Enterprise Deployment Model: Move server solutions between machines using Projects.

✓ Enterprise Automation: Automate everything.*

✓ Enterprise Level Job Control: Define max job time for your jobs.

That wraps up today’s classes...

Course ScheduleTuesday:

Your Data and FME

Wednesday: Automating Everything

Thursday:Expanding Frontiers

FME Workbench Performance Tips & Tricks

A Database for Every Occasion

Working with Attributes in FME

Bulk data processing

Ultimate Real-Time: Monitor Anything, Update Anything

FME in the Enterprise

Remote sensing providers

Web and business data into gold

New + Upcoming in FME Cloud

40-minute talks

Questions?

knowledge.safe.com

Back to FME School - Day 2: Your Data and FME

Software