© 2017 The MathWorks, Inc.€¦ · Run MATLAB scripts on SPARK & HADOOP Master Name Node Worker...

1

2© 2017 The MathWorks, Inc.

MATLAB

Senior Application EngineerThe MathWorks Korea

3

Data Analytics Workflow

Data Analytics• Data Pre-processing• Feature Extraction• Building algorithms, math models• Making business decisions

ata AAAnalllytttiiics

Smart Connected Systems

Business Systems

Analytics Integration • Integrate algorithms with IT • Analytics run on Embedded targets

Data Acquisition• Engineering, Scientific, and Field• Business and Transactional

MATLAB: Single Platform

4

Key Takeaways

1. Distribute applications to non-MATLAB users royalty-free.

2. Integrate MATLAB functions into existing workflows and development platforms.

3. Deploy MATLAB Analytics for Big Data on Hadoop enabled Spark Clusters.

4. Deploy MATLAB applications to service simultaneous user requests enterprise-wide via web or cloud frameworks.

5

Challenges

Multiple internal and external consumers of MATLAB algorithms

Challenging and time consuming to re-code MATLAB algorithms for integration into IT frameworks– Development resources are scarce and time-to-market is short

Company priority to deploy solutions to enterprise scale web or cloud frameworks– Scale application to serve large numbers of simultaneous requests

6

MATLAB Programs Can be Shared With Anyone

Share With Other MATLAB Users Share With People Who do Not Have MATLAB

7

Write Your Programs OnceThen Share To Different Targets

MATLAB

C/C++ExcelAdd-in JavaHadoop .NET

MATLABCompiler

MATLABProduction

Server

StandaloneApplication

MATLABCompiler SDK

Apps Files

Custom Toolbox

Python

With MATLAB Users

With People Who Do Not Have MATLAB

MATLABCoder

Source Code

8

Share with People Who Do Not Have MATLAB

C/C++ExcelAdd-in JavaHadoop .NET

MATLABCompiler

MATLABProduction

Server


MATLABCompiler SDK

Python

Share Applications with No Additional Programming

Integrate MATLAB-based Components With Your Own Software

• Royalty-free Sharing

• IP Protection via Encryption

9

Application Author

End User

1

2

Share Applications Built Completely in MATLAB

MATLAB

ExcelAdd-in Hadoop


Toolboxes

MATLAB Compiler

MATLABRuntimeRuRR3

10

12

1

2

Integrate MATLAB-based Components With Your Own Software

MATLABToolboxes

MATLABRuntime

Application Author

Software Developer

43C/C++

Java

.NETMATLAB

ProductionServer

Python

MATLAB Compiler SDK

13

14

Using MATLAB Compiler SDK to create Python Packages

16

MATLAB and MATLAB Production Serveris the easiest and most productive environment to take your enterpriseanalytics or IoT solution from idea to production

Idea Production

17

Why MATLAB Production Server Matters to You

MATLAB Production Server allow you to continue to work in the environment that you loveNo need to learn another programming languageMATLAB Production Server integrates with enterprise IT infrastructure

MATLAB Production Server integrates MATLAB code into the enterprise IT fabric that you are comfortable withNo need to re-code into another programming languageWeb and cloud friendly architecture

Domain Expert

Solution Architect

18

Scale Up with MATLAB Production Server™

Directly deploy MATLAB programs into production– Centrally manage multiple MATLAB programs and runtime version

s– Automatically deploy updates without server restarts– Most efficient path for creating enterprise applications

Scalable and reliable– Service large numbers of concurrent requests– Add capacity or redundancy with additional servers

Use with web, database and application servers– Lightweight client library isolates MATLAB processing– Access MATLAB programs using native data types

MATLAB Production Server(s)

HTMLXML

Java ScriptWeb

Server(s)

19

MATLAB

MATLABCompiler SDK

Customer examples: Financial customer advisory service

MATLAB Production Server

RequestBroker

AlgorithmDevelopers

RequestBroker

RequestBroker

o Saved € 2 million annually for an external system

o Quicker implementation of adjustments in source code by the quantitative analysts

o Knowledge + MATLAB = Build your own systems

Global financial institution with European HQ

20

MATLAB

MATLABCompiler SDK

Industrial IoT Analytics on AWS


RequestBroker

AlgorithmDevelopers

Industrial Equipment• Networked

communication• Embedded sensors• Data reduction

Business Systems

Users

Global industrial equipment manufacturer

21


RequestBroker

Building Automation IoT Analytics on Azure

Building/HVAC automation control system• Variety of

sensors and controls

• Networked communication

• Data reduction

AzureEventHub

AzureBlob

AzureSQL

MATLAB

MATLABCompiler SDK

AlgorithmDevelopers

Business Systems

Users

Global heavy duty electrical equipment manufacturer

22

MATLAB Production ServerEnterprise Class Framework For Running Packaged MATLAB Programs

Server software– Manages packaged MATLAB progr

ams and worker pool

MATLAB Runtime libraries– Single server can use runtimes fro

m different releases

RESTful JSON interface and lightweight client library (C/C++, .NET, Python, and Java)


MATLABRuntimeR

Request Broker &

Program Manager

EnterpriseApplication RESTful

JSON

EnterpriseApplication

MPS ClientLibrary

23


RequestBroker

&ProgramManager

Enterprise Application

HTTP(S)MWHttpClient

object

Calling Functions

CalculationProcess

CalculationProcess

Worker Pool

24

Databases

CloudStorage

IoT

Visualization

Web

Custom App

Public Cloud Private Cloud

Technology Stack

Platform

Data Business System


Analytics

RequestBroker

AzureBlob

MATLABDistributed Computing

Server

C

25

Example - Integrating with IT systems

WebServer

ApplicationServer

Database Server

Pricing

RiskAnalytics

PortfolioOptimization


MATLABCompiler SDK

Web Applications

Desktop Applications

ExcelAdd-in

26

Development

EnterpriseApplicationDeveloper

Production

MATLABDeveloper

Production Deployment Workflow



Deployable Archive

WebApplication

...

Function Call

MATLAB Algorithm

MATLABCompiler SDK

Initial Test Application

Debug Algorithm

Verify data handling and initial behavior

WebApplication

Deployable Archives

Function Calls

27

Develop and Test with MATLAB Compiler SDK

Test environment for MATLAB Production ServerTest and debug in MATLAB desktop– Details on request transactions– MATLAB debug and profiling with end to end testin

g

e

Application

MATLAB

HTTP

28

Web Management Dashboard – New in R2017a

29

Load Forecasting Demo

Energy load forecasting demoMATLAB Production Server(s)

HTMLXML

Java Script

Web Server(s)

30

MATLAB at Scale


Application server for MATLABFront-end scalabilityManage large numbers of requests to run short-running deployed MATLAB programs

MATLAB Distributed Computing Server

Cluster framework for MATLAB/SimulinkBack-end scalabilitySpeed up computationally intensive programs on computer clusters, clouds, and grids

31

Distinct Offerings Scale Application Access and Computation

Deployed Application

Deployed ApplicationDeployed

ApplicationDeployed ApplicationDeployed

Application

MATLAB Compiler SDK MATLAB Desktop(client)

Parallel Computing Toolbox


GPUGPPU

Multi-core CPU


MATLAB code with batch, parfor, or other

parallel constructs

Request broker

32

Distinct Offerings Scale Application Access and Computation

Deployed Application

Deployed ApplicationDeployed

ApplicationDeployed ApplicationDeployed

Application

MATLAB Compiler SDK MATLAB Desktop(client)

Parallel Computing Toolbox


GPUGPPU

Multi-core CPU

MATLAB code with batch, parfor, or other

parallel constructs

Request broker

MATLAB Compiler SDK

MATLAB Production ServerParallel workers on remote hardware

33

Online Resources

• Documentation – Create and Share Toolboxes

• Website – Desktop and Web Deployment

• Free White Paper – Building a Website with MATLAB Analytics

• Website – Using MATLAB With Other Programming Languages

34

Supplemental Slides

Use the following slides for more detailed discussions on various implementations using MATLAB Production Server.

46

Challenges of Big Data

“Any collection of data sets so large and complex that it becomes difficult to process using … traditional data processing applications.” (Wikipedia)

Rapid Data Exploration

Scalable Algorithms

Integrate Big Data ApplicationsVisualize

Develop

Deploy

47

DatastoreHDFS

Hadoop: The Big Data Platform

Reduce

Node

Node

Node Data

Data

Data

Map

ReduceMap

ReduceMap

Map ReduceR

Map

MapM

ReduceR

ReduceR

48

Datastore

Matlab Integration with Hadoop clusters

map.mreduce.m

HDFS

Node Data

Node Data

Node Data

Map ReduceRe

Map ReduceRe

Map ReduceRe

49

Deploy Applications with Hadoop

Compile MATLAB Map ReduceCode

Datastore

HDFS

Node Data

Node Data

Node Data

Map ReduceRe

Map ReduceRe

Map ReduceRe

MATLABruntime

50

Use MATLAB with Spark on Gigabytes and Terabytes of Data

tall arrayor

tall tables

51

Run MATLAB scripts on SPARK & HADOOP

Worker NodesMaster Name Node

Hadoop & Spark Library

HDFSYA

RN

Data NodesResourceManager

Edge Node

Spark-submit script

Job submitted using Java RDD API

MATLAB workers on worker nodes in the cluster• MDCS workers (working from MATLAB)

52

Example: Running on Spark enabled Hadoop

%% Define the Execution Environment.% Hadoop/Spark Cluster setenv('HADOOP_HOME', '/dev_env/cluster/hadoop');setenv('SPARK_HOME', '/dev_env/cluster/spark');

numWorkers = 16; cluster = parallel.cluster.Hadoop;cluster.SparkProperties('spark.executor.instances') = num2str(numWorkers);mr = mapreducer(cluster);

% Access the datads = datastore('hdfs://hadoop01:54310/datasets/taxiData/*.csv');tt = tall(ds);

% Define the Execution Environment.% Desktopmr = mapreducer(gcp);

% Access the data.ds = datastore(‘C:/datasets/taxiData/*.csv');tt = tall(ds);

Desktop Code

Spark + Hadoop Code

Hadoop Access

Spark Connection

Cluster Config for Spark

PCT, Datastore, tall

53

Example: Running on Spark and Hadoop

54

Run MATLAB scripts on SPARK & HADOOP

Worker NodesMaster Name Node

Hadoop & Spark Library

HDFSYA

RN

Data NodesResourceManagerEdge Node

MATLAB workers on worker nodes in the cluster• MATLAB Runtime (deployed applications)

Compile MATLAB CodeC

55

Deploying Spark Applications

Date post:	12-May-2020
Category:	Documents
Upload:	others
View:	11 times
Download:	0 times

© 2017 The MathWorks, Inc.€¦ · Run MATLAB scripts on SPARK & HADOOP Master Name Node Worker...

Documents