Post on 03-Oct-2020
transcript
EdgeBench: Benchmarking Edge Computing Platforms
Anirban Das*, Stacy Patterson*, Mike P. Wittie^*Department of Computer Science, RPI
Networked Systems Lab^Gianforte School of Computing, Montana State University
12/20/2018
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms2
• Motivation
• Related Work
• System Architectures
• Pipelines and Applications/ Workloads
• Experimental Results
• Conclusion
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms3
Cloud Services
Edge Devices
Edge Devices
Edge Devices
Sensors / Data sourcesSensor data Data flow
● Partial or Full computation ● Manage and Deploys
Edge Modules ● Provide connectors to
other services
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms4
• Ubiquitous intelligence• Speech detection • Image recognition• Sensor data stream• Autonomous cars• Augmented Reality
Ref: https://www.gartner.com/smarterwithgartner/5-trends-emerge-in-gartner-hype-cycle-for-emerging-technologies-2018/
5-10 years
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms5
Cloud Services
Edge Devices
Edge Devices
Edge Devices
Sensors / Data sourcesSensor data Data flow
● Partial or Full computation ● Manage and Deploys
Edge Modules ● Provide connectors to
other services
Serverless
Serverless
AWS Greengrass
Azure IoT Edge
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms6
Big 4: AWS Lambda, Azure Functions, GCF, IBM Openwhisk• CPU intensive benchmarks using Serverless and Hyperflow
▪ Malawski et. al., 2017• Azure based prototype for performance oriented serverless and measures
performance using custom made tool▪ McGrath and Brenner, 2017
• Propose a micro benchmark for cost and performance modeling▪ Back and Andrikopoulos, 2018
• Provides a real world example of running k-Means clustering on AWS Lambda▪ Deese, 2018
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms7
• Need to compare vendors in Edge Computing• Need to compare edge architectures with cloud only architectures• Feasibility of edge architectures
• Contributions:▪ Developed benchmark EdgeBench▪ Developed benchmarking methodologies and metrics of interest▪ Developed applications based on real world use cases▪ Studied two platforms / industry vendors:
− AWS Greengrass− Microsoft Azure IoT Edge
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms8
AWSGreengrass
AzureIoT Edge
S3
DynamoDB
Lambda
Azure Blob Storage
Azure Stream Analytics
Event Hubs
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms9
• Canonical applications from real-world use cases
▪ Scalar Sensor Emulator:
− Extremely light-weight workload - A scalar sensor value generator
▪ Image Classification:
− A representative workload from the image processing/ classification domains like autonomous cars, AR
▪ Speech to Text Decoding/Translation:
− An edge use-case of speech to text decoding inspired from the popularity of Amazon Echo and Google Home
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms10
AWSGreengrass
AzureIoT Edge
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms11
Image Classification/ Object Recognition• Python • MXNet framework (Squeezenet)• Workload: Imagenet 2012 dataset
Speech to Text• Python • PocketSphinx: Python Port:
(https://github.com/bambocher/pocketsphinx-python)• Workload: Samples from Tatoeba Database from Mozilla Common Voice
platform
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms12
• 3 UTC timestamps:▪ T1 at the edge▪ T2 at IoT Hub▪ T3 at S3/Blob
• Feasibility of edge device▪ Compute time▪ Memory and CPU utilization
• Feasibility of applications▪ Time in Flight / Flight time▪ End to End Latency
• Bandwidth Savings▪ Payload Size
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms13
• Files send sequentially (10-15 s delay)• Lambda memory at 3008 MB and Azure Consumption Host Plan• Metric: End to End Latency
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms14
• Raspberry Pi 3B• TM 2000A Stratum 1 for time sync• Azure and AWS locations US East
North Virginia• Local Lambda Long running• GGC Core 1.5.0, Azure IoT Hub
device client 1.4.0.
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms15
● Image Recognition Sub second, Audio slow
6s4.77s
.7s.47s
negligible
Edge Only Pipelines
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms16
1.1s.87s
≈ 95sSlowest, due to batching
Fastest
Both Edge and Cloud
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms17
• Massive reduction in BW usage in cloud vs edge pipelines:
▪ AWS: 36x in audio and 81x in image▪ Azure: 36x in audio and 77x in image
• Average single payload size for edge apps:
Image: 752 bytes Audio: 162 bytes Scalar: 234 bytes
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms18
• Image Pipeline: 1 traffic camera, image every 10 second for 1 month• Input data size : 259,200 x 143 KB• Cloud Config: 3008 MB Lambda
• Cost:▪ Greengrass : ≈ 1.56 USD / month▪ AWS Cloud Solution : ≈ 8.027 USD / month
• Cloud solution 5.3x more expensive at least.
• Data Transfer:▪ Greengrass : 253 MB ▪ AWS Cloud Solution : 35.4 GB
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms19
• Presented EdgeBench (https://github.com/akaanirban/edgebench)▪ Methodologies, Applications, Performance on Greengrass and Azure IoT Edge
• Our results show:▪ Performance on the edge comparable for both platforms▪ Cloud is faster than edge▪ Bandwidth saving is massive using edge architectures
• Is one platform better than the other? ▪ Depends on use case for e.g., batching vs event based
• Future work: ▪ Expanding into Google and IBM’s products▪ Expand study with different model sizes and applications▪ Standardize deployment procedure (open problem)
− Need for frameworks like Serverless for homogeneity− Greengo for Greengrass (https://github.com/dzimine/greengo)
Thank You
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms21
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms22
≈ 200 MB88-90%
negligible
Edge Only Pipelines
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms23
AWS Greengrass Azure IoT Edge
Runtime Python 2.7, Node.JS 6.10, Java 8, C, C++ C#, C, Node.JS ver > 0.4.x.x, Python (both 2.7 and 3.6), and Java 7+
Deployment Method Lambda Functions● Greengrass Containerized Non Containerized (as of
ggc core 1.7)● Install heavier libraries directly on Raspberry Pi
Docker Containers● Orchestrated using Moby● Can package anything in Containers
Triggers Routes available
15 (e.g. S3, Dynamo DB, Lambda, Cloudwatch logs, SNS, Step Functions etc.)
4 (e.g.Blog Storage, Event Hub, Service Bus Queue, Service bus topic ) (Can directly deploy Azure ML models and ASA jobs into IoT Edge)
Parallel Execution Parallel Lambdas can be triggered to run locally N/A
Deployment boto3, aws-cli azure-cli, VSCode
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms24
12/20/2018
Networked Systems Lab, RPI
EdgeBench: Benchmarking Edge Computing Platforms25