Date post: | 21-Jan-2018 |
Category: |
Software |
Upload: | safe-software |
View: | 1,312 times |
Download: | 0 times |
Back to FME School3-Day webinar series
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
Let’s begin today’s lesson...
Potent Potions for Batch Data
Processing
The Heavy Lifters: Two key Transformers for Bulk Data Processing
Meet the
FeatureReader & FeatureWriter
FeatureReader (since FME 2011)
Solves problems such as:
1. Read data within a selected boundary
2. Read data within a user drawn area3. Process a directory of files
4. Read in data FME has written in the same workspace
FeatureReader
FeatureWriter (since FME 2016)
Do things AFTER the data is written in FME Workbench.
Less need for Shutdown Python or TCL
● copy data after writing● FTP results● Email an attachment● Load into Dropbox or S3● Insert data into HTML file● Database validation● FME Server Notification● 3rd Party Integration● Can Write in Parallel!
FeatureReader and WriterIn the Real World
Web Client with FME
FeatureReader & WriterIn the Real World
FeatureWriter Demo
Now Let’sBatch...
250,000 CAD files & rasterson mobile devices
Tip: Know Your Potions and Choose Wisely
Today’s Potions1. Wildcards
2. Batch Deploy
3. Parent/child Workspaces
4. Parent/child Server Workspaces
Potion 1: One Wild<card>
Dataset
Multi-Dataset Picker
Multi-Dataset Picker
Multi-Dataset Picker
Multi-Dataset Picker
ShapefileMapInfo
Most rasters
DWGDGNSQLite
Dataset Wildcards
Extended glob syntax:
Symbol Matches
? Any single character
* Any sequence of zero or more characters
[chars] Any single character in chars.
[a-d] Any character between a and d inclusive
{a,b,...} Any of the sub-patterns a, b
/**/ 0 or more subdirectories
Time to brewPotion 1
Wildcard Bulk Data Processing Enticements
✓ Simple to set up
✓ Can transform across file boundaries
- Needs memory & time
Wildcard Bulk Data Processing
Pitfalls
x Recovery from data errors difficult
x Feature Type vs File vs Format Issues
x No granular logx No ability to
parallelize
Potion 2: Batch Deploy
Batch Deploy Script Writer
Batch Deploy Script Writer
Batch Deploy Script Writer
Batch Deploy Script Writer
Batch Deploy Script Writer
Batch Deploy Script Writer
Time to brewPotion 2
Batch Deploy Enticements
✓ Simple to set up✓ Runs quickly✓ Can script via
command line✓ Run on demand
Batch Deploy Pitfalls
x Recovery from data errors difficult
x No granular logx Destination dataset
naming can be tricky
Potion 3: Parent/ChildWorkspaces
Parent/Child Workspace Ingredients
• Parent Workspace:– PathReader– WorkspaceRunner
• Child Workspace:– FeatureWriter
Parent Ingredients
Parent Ingredients
Child Ingredients
Time to brewPotion 3
Parent/Child Workspace Enticements
✓ Separate transformation from workflow
✓ Generate audit logs✓ All authored within
Workbench
Parent/ChildWorkspace
Pitfalls
x Not all writers can be used concurrently
x Slow to run each child workspace separately
x Recovery from data errors not easy if concurrent runs used
Potion 4: Parent/Child Server
Workspaces
Parent/Child Server
Workspace Ingredients
• Parent Workspace:– PathReader– FMEServerJobSubmitter– FMEServerJobWaiter
• Child Workspace:– FeatureWriter
Parent Ingredients
Parent Ingredients
Parent Ingredients
Child Ingredients
Time to brewPotion 4
Parent/Child Server
Workspace Enticements
✓ Separate transformation from workflow
✓ Generate audit logs✓ All authored within
Workbench✓ Make full use of
parallelism = FAST
Parent/ChildServer
Workspace Pitfalls
x Not all writers can be used concurrently
x Data needs to be accessible to Server Engines- Consider using Server Data Resources
x Craft your reload/audit plan
Summary● Many ways to handle bulk data moves
● Choose your potion wisely - each has pluses and minuses
● FeatureReader/FeatureWriter increase flexibility
● FME Server is the most robust automation choice
Ultimate Real-Time Monitor Anything, Update Anything
FME Server Notification Service lets you act on events as they happen, and send information as it becomes available.
Publications and Subscriptions let you publish to and monitor different systems and data.
NotificationsWhat they are for
What they are not for
NotificationsWhat they are for
✓ A brief message, usually to trigger an action.
What they are not forx Transmitting large amounts
of spatial data.
NotificationsWhat they are for
✓ A brief message, usually to trigger an action.
✓ Triggering an FME Server response to an event that happened outside of FME.
What they are not forx Transmitting large amounts
of spatial data.x Triggering an FME Server
response to a continuous series of messages (many per second).
NotificationsWhat they are for
✓ A brief message, usually to trigger an action.
✓ Triggering an FME Server response to an event that happened outside of FME.
✓ Sending a message about something that happened in FME.
What they are not forx Transmitting large amounts
of spatial data.x Triggering an FME Server
response to a continuous series of messages (many per second).
x Sending more than one message per second about what’s happening in FME.
FME Server Notification Service
See the list of what built-in publications and subscribers can be: http://fme.ly/protocols
Workflows for“Ultimate Real-Time”
Poll it Push it Update it
Poll it“Are we there yet?”
Polling Workflow1. Change detection
○ GeoRSS reader○ Caching○ Timestamps
2. Run the workspace on a scheduleTimestamp
Scheduling (FME Server)
GeoRSS Reader
GeoRSS Reader
• Contains change detection logic
• Option to only read new features
Caching
• Whole datasets or timestamps
• ChangeDetector
• SQL - match on ID
ChangeDetector Example
SQL to find different features:@Value(uuid) = "uuid" AND @Value(aqi) != "aqi"
SQL Example
Timestamps
● Cache the timestamp
● Last updated attribute
● Filter features newer than timestamp
● Add a timestamp attributes to your dataset
Reads last cached time and replaces with current time
Checks against timestamp and filters out old features
Pros and ConsCaching whole datasets
✓ Easy to set up and detect changes.x Need to store entire
dataset - processing time?
Caching timestamps
✓ No storage needed.x Harder to set up -
variables, datetimes.
Consider: Does my dataset have a timestamp? Can I rely on the timestamp to indicate change?
Story: Real-Time Traffic
✓ Poll Waze feed
✓ Parse XML
✓ Filter
✓ Transform
✓ Validate
✓ Email notifications
Workflows for“Ultimate Real-Time”
Poll it Push it Update it
Push It
Pushing Methods• Database triggers• Webhooks
In FME Server:• Direct URL• REST API
Database Triggers
● Databases can trigger events.
● Functions with HTTP requests can push data to FME Server.
Database Triggers
Webhooks
● i.e. Give systems/services a URL that will respond.
● Send the notification to FME Server.
● Queue the received notifications and process the requests on a separate thread.
Pros and ConsDatabase Triggers
✓ Pushes right to FME Server topic.
✓ Can push lots of changes to a holding table.
✓ Perfect for real-time database changes.x Database permissions.
Webhooks
✓ No need to waste effort polling.
✓ Truly real time.x Complex to set up.x Not all systems provide
or accept webhooks.
Direct URLSystems can use HTTP to push data to topics or trigger workspaces to run.
Topic POST URL
REST APIIn-depth interaction
with FME Server
● Send/receive notifications● Run jobs● Manage connections● Manage users● Licensing● Backups● Scheduling● Etc!
Workflows for“Ultimate Real-Time”
Poll it Push it Update it
To Poll or Push?
Poll vs. Push Considerations
● Simplicity vs. “real-time”ness● Can the system push to FME Server?● Are jobs too long for polling?● API restrictions for polling?
Tip: Use FME Server for internal systems and FME Cloud for external systems.
Update It
Updating Methods
• FME Writerso Bulk
drop/load/overwriteo Update option
• SQLExecuter• HTTPCaller
FME Writers: UPDATE operation
INSERT, UPDATE or DELETE in the middle of a workflow.
POST data or changes to systems.
Summary
• Polling workflow:o Change detection + scheduling
• Pushing options:o Database triggers, webhookso FME Server: URL or REST API
• Updating:o Operations in writer
parameterso SQLExecutoro HTTPCaller
Questions?
Database triggers tutorial:fme.ly/dbtriggers
Webhooks etc:fme.ly/web
REST API documentation:fme.ly/restv3
FME Server: Charms for the
Enterprise
Enterprise-Level Charms
★ Security - a new level with new charms
★ Deployment - move solutions easily
★ Automate the enterprise
★ Jobs Control
Security Charm #1Active Directory
✓ Any number of Active Directory domains✓ Any number of users per domain✓ A hybrid model - mix server and AD users
Active Directory
Functionality based on feedback and experience from users! Thank you!
Active Directory Support
Import Active Directory
Users
Hybrid security model
Over to FME
Security Charm #2
Sharing and user control
✓ Users can share with other users or groups.
✓ Sharing defined for everything except “shared resources”.
✓ Users are isolated from each other.
✓ No load on Administrator.
Familiar Sharing Model (Google, Dropbox, etc.)
Modern Sharing Model
User-Controlled Security
✓ Users have control of all FME Server components
✓ Similar to state-of-the-art services like Google Drive.
Security Charm #3Connections & 3rd-Party Applications
Connection Based 3rd Party Integration
Why use connections?
● Encapsulate user credentials in a secure manner.
● Define for Databases and Web Applications
● Share across multiple workspaces.
● Share without revealing credentials.
Connection Based 3rd Party Integration
Many web connections.● Amazon Web Services● OAuth 2.0 (tens of
thousands of services)● Token● HTTP Authentication
All FME-supported databases.
Demo
Database Connections
Web ConnectionsMany web connections are shipped with FME
Web Connections
More are being added toFME Hub all the time
Security Charm #4We secure your infrastructure too.
On Premises vs FME Cloud
On-premise:
● you secure your infrastructure
● you configure FME Server security
FME Cloud:
● you configure FME Server security
FME Cloud SecurityWe manage data, application and transmission security.
● HTTPS only
● Annual security audit
● Automated security patching
● Leverage AWS Compliance - Industry leaders
● Control firewall rules (protocols & ports)
Deployment CharmEasily move solutions
Challenge: How can you move an FME Server
solution from one server to another?
Your FME Server solutions usually consist of many components.
Workspaces Subscriptions Schedules
Publications Resources Connections
Custom Formats Custom Transformers Topics
FME Server Projects
Portable Server SolutionsPlace your screenshot here
FME Server ProjectsPortable way to move, distribute, and share FME Server solutions.
Benefit #1 - Move a project from one machine to another.
Development StagingTest Production
Projects.
Benefit #2 - Backup (export) a project any time for archiving.
FME Server Projects
Projects.
Benefit #3 - Share your projects.
First - Export your project!
Projects.
Project Export
FME Server Projects
Projects.
Benefit #3 - Share your projects
Next - Send to those with whom you want to share it.
Projects.
Project Export
FME Server Projects
Projects.
Benefit #3 - Share your projects Finally - Receivers simply import the project.
Projects.
Project Export
FME Server Projects
Projects.
Benefit #4 - Publish Projects to FME Hub. (Coming Soon)
Share FME Server solutions with the FME community.
Projects.
Project Export
FME Server Projects
Automation Charm
EnterpriseAutomation
This charm is so powerful that we have another presentation
just on this.
Termination Charm
Goal: Ensure FME Engines don’t get stuck on a job.
Workflows are often dependent on 3rd Party applications/services.
An engine may hang or take too long waiting on responses.
For any job you now have a new setting (Schedules and run Workspace)
Max elapsed time in seconds job will run.If time expires job is killed!
Charm: Specify maximum time a job can run.
Charm: Specify maximum time a job can run.
If job takes longer than allocated then the job is cancelled.
Summary - FME Server 2017 Charms✓ Enterprise Level Security: Enable fine-tuned control with
the new security model.
✓ Enterprise Deployment Model: Move server solutions between machines using Projects.
✓ Enterprise Automation: Automate everything.*
✓ Enterprise Level Job Control: Define max job time for your jobs.
That wraps up today’s classes...
Course ScheduleTuesday:
Your Data and FME
Wednesday: Automating Everything
Thursday:Expanding Frontiers
FME Workbench Performance Tips & Tricks
A Database for Every Occasion
Working with Attributes in FME
Bulk data processing
Ultimate Real-Time: Monitor Anything, Update Anything
FME in the Enterprise
Remote sensing providers
Web and business data into gold
New + Upcoming in FME Cloud
40-minute talks
Questions?
knowledge.safe.com