RAIN RFID - A world of business & consumer connectivity - Real … · 2020-01-04 · RFID Reader...

transcript

Real-time RAIN Data in the CloudBy: Colin BookmanJune 9th, 2016

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Proprietary + Confidential

Realtime Inventory

70%Of the items seen in inventory

systems are actually where they

claim to be. We've found this to

hold true for retail and

distribution centers.

Where are my items What do I need to do How do I get this data

Performing quick lookups of what

physical locations have a particular

item, and what items are in a particular

location.

What items need to be moved from

one location to another. How many of

each item needs to be moved, and to

where.

My Current inventory system onlysupports a count, not EPC level data.My inventory system cannot handlemany minor adjustments of inventorydata per second.

Problems Faced

Key GCP Technology

Pubsub

Storage

NoSQLBlob SQL

CloudStorage

CloudBigtable

CloudDatastore Cloud SQL

Good for:Structured and unstructuredbinary or object data

Such as:Images, large media files,backups

Good for:Getting started,App Engine,serve use cases

Such as:User profiles,product catalog

Good for:Web frameworks,existing applications

Such as:User credentials, customerorders

Good for:Heavy read +write, events, andanalytical data

Such as:AdTech, Financialand IoT data

Bigtable

Read/Write Latency at 99% (ms)

BigTable

Dataflow

BigQuery

Kubernetes

kubelet

kubeletCLI

users master nodes

kubelet

scheduler

controllers

apiserver

Overwhelmed,So am I

Realtime Inventory

Low Latency Simple Stupid onsiteequipment

Real Time InventoryIngestion & Aggregation

Ideally takes server less than 100ms

to prepare a response to an api

request

Equipment will be in remote hard to

get places. Keeping it simple stupid

makes support easier.

Inventory data should be ingested bythe system to as close of real-time aspossible. Ideally no cron jobs tocalculate inventory counts.

Technical Requirements

http://www.dailyhaha.com/_pics/failure-dog.htm

Failures

Attempt #1, Firebase

RFIDReader

● Fixed readers all over the environment

● Readers performing a Session 1 Target AB read

Ingestion ● RPI ingests RFID data, and pushes it to Firebase

Application ● Webapp queries Firebase for all the EPC tags and calculates item counts

Attempt #1, Firebase

Attempt #2, MySQL

RFIDReader

● Fixed readers all over the environment

● Readers performing a synchronized Session 1 Target AB Session 3 Target A read

Ingestion ● RPI ingests RFID data, and pushes it to Firebase AppEngine

● AppEngine writes each tag into MySQL

Application ● Webapp queries Firebase AppEngine for EPC tags Gtin14 Counts

Attempt #2, MySQL

EPC Location Gtin14 LastSeen

3018E511C735D2C0000000EA MTV.CYD1.Hanger1 12345678901231 2016-06-09T01:00:00+00:00

3018E511C735D2C0000000EB MTV.CYD1.Hanger1 12345678901231 2016-06-09T01:00:00+00:00

3018E511C735D2C0000000EC MTV.CYD1.Hanger3 12345678901231 2016-06-09T01:00:00+00:00

3018E511C735D2C0000000ED MTV.CYD1.Hanger2 12345678901231 2016-06-09T01:00:00+00:00

MySQL Schema

Attempt #2, MySQL

Attempt #3, Datastore (NoSQL)

RFIDReader

● Fixed readers / Phased Arrays all over the environment

● Readers performing a Session 3 target A read

Ingestion ● RPI ingests RFID data, and pushes it to AppEngine Pubsub

● AppEngine pulls pubsub messages and writes each tag into MySQL Datastore

Application ● Webapp queries AppEngine for Gtin14 Counts

Key Tags LastUpdated

<ReaderId> EPCSGTIN96[] <UTC Time>

Google.MTV.Foo [3018E511C735D2C0000000EB,

3018E511C735D2C0000000EC,

3018E511C735D2C0000000E0]

2016-06-

09T01:00:00+00:00

Raw RFID Tag Documents

Key Count LastUpdated

Google.MTV.Foo.12345678901231 10 2016-06-

09T01:00:00+00:00

What worked

Attempt #4, Dataflow + Datastore + BigQuery

RFIDReader

● Fixed readers / Phased Arrays all over the environment

● Readers performing a Session 3 target A, followed by a shorter Session 3 Target B read

Ingestion ● RFID Reader ingests RFID data, and pushes it to Pubsub

● A sliding window dataflow job processes tags and outputs to a pubsub topic `inventory-

aggregation` aggregation counts, which app engine ingests and pushes to datastore.

● Streaming inserts to bigquery also process the raw rfid tag stream

Application ● Webapp queries AppEngine for Gtin14 Counts

Dataflow + Datastore + BigQuery

RFID ReaderCapture RFID Data

Physical Location

Cloud Pub/SubStreaming Changes

Messaging

Cloud DataflowAggregation

Processing

Cloud DataStoreInventory Store

Events

Cloud BigQueryData Warehouse

Analytics

App EngineInventory APIs

Services

Attempt #4, Dataflow + Datastore + BigQuery

Future Plans

RFIDReader

● Fixed readers / Phased Arrays / Mobile Readers all over the environment

● Fixed Readers performing a Session 3 target A, followed by a shorter Session 3 Target B read

Ingestion ● RFID Reader ingests RFID data, and pushes it to Pubsub

● A dataflow job pushes the raw RFID data to bigtable and bigquery. It also analyzes the rfid data

on a sliding window and writes the results to bigtable and bigquery.

Application ● Webapp queries AppEngine Managed VMs for Gtin14 Counts

Dataflow + BigTable + BigQuery

RFID ReaderCapture RFID Data

Physical Location

Cloud Pub/SubStreaming Changes

Messaging

Cloud DataflowInventory Streams

Processing

Cloud BigTableInventory Store

Events

Cloud BigQueryData Warehouse

Analytics

App EngineInventory APIs

Services

Row Key EPC: POSCounts

<Location>|<Grouping>.<Gtin14> <EPCSgtin96> 3019C56A40A1288000000002 Square

LastSeen

Latitude

Longitude

ReaderType

lastseen: 2016-06-09T01:00:00+00:00

latitude: 37.422428

longitude: -122.084329

readerType: fixed

BigTable Schema

http://cdn-d4d.kxcdn.com/wp-content/uploads/2015/02/4.jpg

Security

PKI Each RFID Reader has its own Public and Private key pair

TLS All of our cloud endpoints use http over TLS

IAM Cloud services use Identity Access Management tolimit access to minimal permission set

Logs Store logs for post mortem analysis

http://treasure.diylol.com/uploads/post/image/222172/resized_the-most-interesting-man-in-the-world-meme-generator-i-have-a-question-for-you-what-do-you-want-to-ask-me-46a76c.jpg

Questions