Business Process Automation with Google App Engine and Google Data APIs

Automating Business with Google APIs

David Hodge

Thursday, March 7, 13





We automate business processes using cloud computing and

machine learning.

Dave & Bediako


IAAS and PAAS

- Amazon, Google, Microsoft

- DevOps, NoOps, OOPs

- Beware single point of failure - read SLAs


Google Docs Drive

• Google Docs API deprecated

• Google Drive Polyglot API - .NET, Java, Dart, Go, GWT, Java, Javascript, Objective-C, PHP, Python, Ruby

• ReSTful API - create and manage documents


Google Spreadsheets API

• Spreadsheets, Worksheets, List Row, Cell

• Create, Delete, Share

• List Based Feeds vs Cell Based Feeds

• Updating multiple cells with Batch


Google Spreadsheets API

• Gotchas

• Queries - age > 25 and height < 175


Google Apps Scripts

• Javascript cloud scripting language

• Automate workflows

• Customize spreadsheets

• Schedule Tasks

• Save data using Spreadsheets, JDBC, ScriptDB


Google App Engine


Google App Engine

Backends


Google App Engine

Backends

Storing Data


Google App Engine

Backends

Storing Data

Blobstore


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage

Capabilities Channel Endpoints


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail

Memcache


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail

Memcache Multitenancy


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail

Memcache Multitenancy OAuth


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail


Prospective Search


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail


Prospective Search

Search


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail


Prospective Search

Search

Task Queues


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail


Prospective Search

Search

Task Queues

Users


Google App Engine

Backends

Storing Data

BlobstoreApp

Identity

Google Cloud

Storage


Images Logs Mail


Prospective Search

Search

Task Queues

Users

XMPP


One does not simply

Automate using Google APIs


How we build softwareA Customerhas a nick namehas a full namehas a date of birthhas a social security number which is sensitivecan be viewed or created or changed by a customer service representative

An Orderbelongs to a Customerhas a descriptioncan be viewed or created or changed by a customer service representative

An Order Line Item is auditedbelongs to an Orderhas a descriptionhas a price which is a numbercan be viewed or created or changed by a customer service representative

Dictation + Airlift on Google App Engine


Current processes manual and paper based for recording line items for accounts receivable and accounts payable.


Customer Widgets Spreadsheets


Widget Maker Spreadsheets


Evolution One - Cron<?xml version="1.0" encoding="UTF-8"?><cronentries> <cron> <url>/accounting/receivable</url> <description>Process widgets sold for accounting line items</description> <schedule>every day 12:30</schedule> <timezone>America/New_York</timezone> </cron> <cron> <url>/accounting/payable</url> <description>Process line items for widget making workers</description> <schedule>every day 03:30</schedule> <timezone>America/New_York</timezone> </cron></cronentries>


Evolution Two - Backends

• Use Backends to process long lived requests - No Request Deadlines

• Higher CPU up to 4.8GHz and Memory limits - up to 1GB

• Also using Spreadsheet Batch



• Dynamic vs Resident - Dynamic Backends become alive only on request and are turned off when idle



<backends> <backend name="accounting"> <options> <dynamic>true</dynamic> <fail-fast>true</fail-fast> <class>B8</class> </options> </backend></backends>

backends.xml

<?xml version="1.0" encoding="UTF-8"?><cronentries> <cron> <url>/accounting/receivable</url> <description>Process widgets sold for accounting line items</description> <schedule>every 20 minutes</schedule> <timezone>America/New_York</timezone>

<target>accounting</target> </cron> <cron> <url>/accounting/payable</url> <description>Process line items for widget making workers</description> <schedule>every 40 minutes</schedule> <timezone>America/New_York</timezone>

<target>accounting</target> </cron></cronentries>

cron.xml


Task Queues work discretely on a unit of work

Push - processes tasks based on rate configuration, auto scales

Pull - allows for specific time and timeframe to run


Task Push Config<queue-entries> <queue> <name>fooqueue</name> <rate>1/s</rate>

<bucket-size>40</bucket-size> <max-concurrent-requests>1</max-concurrent-requests> <retry-parameters> <task-retry-limit>7</task-retry-limit> <task-age-limit>2</task-age-limit> </retry-parameters> </queue> <queue> <name>barqueue</name> <rate>1/s</rate> <retry-parameters> <min-backoff-seconds>10</min-backoff-seconds> <max-backoff-seconds>200</max-backoff-seconds> <max-doublings>0</max-doublings> </retry-parameters> </queue> </queue-entries>

queue.xml


Task Queue Headers

• X-AppEngine-QueueName, the name of the queue (possibly default)• X-AppEngine-TaskName, the name of the task, or a system-generated

unique ID if no name was specified• X-AppEngine-TaskRetryCount, the number of times this task has

been retried; for the first attempt, this value is 0. This number includes attempts where the task failed due to a lack of available instances and never reached the execution phase.

• X-AppEngine-TaskExecutionCount, the number of times this task has previously failed during the execution phase. This number does not include failures due to a lack of available instances.

• X-AppEngine-TaskETA, the target execution time of the task, specified in microseconds since January 1st 1970.


Task Push Queues

import com.google.appengine.api.taskqueue.Queue;import com.google.appengine.api.taskqueue.QueueFactory;import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*;

//Discrete job for processing single widget to write customer line item Queue queue = QueueFactory.getDefaultQueue(); queue.add(withUrl("/widgetorderforcustomerjob").param("widgetKey", key));

//Discrete job for processing single widget to write vendor line item queue.add(withUrl("/widgetorderforvendorjob").param("widgetKey", key));


Google DocsGoogle App Engine

Automation Achieved

Receivable Line Items Job

Payable Line Items Job

Google Spreadsheet

Line Items Per Client

Google Spreadsheet

Line Items Per Vendor

Invoices

Payments


Experiment

EvolveArchitecture


Support

• Google Plus Hangouts

• Google Forums

• Fellow Developers


Some More Tips

• Python: Bulk Loader Tool to download/upload data - good for any flavor of GAE

• Get involved - NoOps doesn’t mean you don’t have to know your technology stack

• https://github.com/LucidTechnics/google-apps-util

• https://code.google.com/p/googleappengine/wiki/WillItPlayInJava


https://github.com/LucidTechnics/google-apps-util




https://code.google.com/p/googleappengine/wiki/WillItPlayInJava




“Using no way as a way, having no limitation as limitation.”

- Bruce Lee


Official White House Photo by Pete SouzaThursday, March 7, 13

David Hodge@davezen1


Date post:	16-May-2015
Category:	Documents
Upload:	david-hodge
View:	1,039 times
Download:	0 times

Business Process Automation with Google App Engine and Google Data APIs

Documents