© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Intelligent DataOps powered
by Pentaho
IoT Insights and
Outcomes
Pol HuyzentruytSenior Solutions Consultant, Hitachi Vantara Belux
October 2019
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Agenda
Introduction to Hitachi and Intelligent DataOps
Accelerate your journey to DataOps with Pentaho Data Pipelines
Use cases
Q & A
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
100+ years of experience
The Hitachi Story
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Your Data. Your Innovation.Digital Unlocks the Potential of
Data to Drive Business Outcomes
Reduce thefts,
improve safety,
lower costs
Reduce Costs
Offer your
customer new
services in order to
enlarge your
business
New Revenue
Models
Increase asset
utilization, improve
employee
effectiveness
Increase
Efficiency
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
The Customer Challenge
Getting the right data, to the right place at the right time is challenging for most
organizations. Here’s why:
Data is more diverse,
than ever before
Data is more
distributed than ever
Data is siloed across
applications, clouds, and
internal tools
It’s difficult for
organizations to know
what data they have
and where it is
Discovering and preparing
the right data for analytics
can be slow and manual
Lack of lean and agile
methodologies across
the data value chain
It’s difficult to reduce
costs in the way data is
stored and managed
The process of
transforming data into
value is difficult to repeat
Governance and
compliance requirements
are growing
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Data Journey
Leaky Pipe Analysis
Value of DataPotential Value
IN
Value of DataActual Value
OUT
Growth of data complexity, coupled with customer transition to hybrid multi-cloud,
creates major challenges in data integration, orchestration and analytics
Still Heavy IT Involvement
Only 40% claim to provide
self-service
Unstructured Data
< 1% of unstructured
data analyzed.
80% Rule
Data prep is still 80% of an
analytics project
Deploying ML models is Hard
66% do not have automated
process
Skills gap with Big Data tools
28k Jobs on LinkedIn with
“Spark”
Cloud Creating Data Siloes
Average 5 clouds per
organization
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Intelligent DataOps – the concept
Acquire the Data Store the Data Share the Data
Developers
Data Scientist
Data Engineers
Data Analysts
Business Users
Right Data. Right Place. Right Time.
STREAMING DATA
UNSTRUCTURED DATA
STRUCTURED DATA Enrich the Data
Explore the Data
“Data Delayed is Data Denied”
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Ingestion Profiling Enrichment Blending AI / Orchestration
Discovery
Edge-to-Cloud Data Pipelines
Administration SecurityLifecycle
ManagementScalability Lineage Monitoring Automation
Data Analyst/
Steward
Business
AnalystData
EngineerData Scientist /
Data Ops
Data
EngineerData Analyst/
Steward
Data Insights
Pentaho Data
Integration
Pentaho Business Analytics
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Intelligent DataOps and Pentaho
Acquire the Data Store the Data Share the Data
Developers
Data Scientist
Data Engineers
Data Analysts
Business Users
Right Data. Right Place. Right Time.
STREAMING DATA
UNSTRUCTURED DATA
STRUCTURED DATA Enrich the Data
Explore the Data
Pentaho Data
Integration
Pentaho Business Analytics
“Data Delayed is Data Denied”
Big Data Analytics ArchitectureIntegrating The Analytics & IoT World
Location
Web
Social
Media
Network
EDW Data
Marts
Customer
Orders
Financials
PDIPDI
Pentaho
Pentaho
Hadoop
Cluster
NoSQL
Analytic
DB
PDI
Pentaho
On-Demand Integration & Blending
PDIPentaho
PDIPentaho
HDInsight
Machine Learning
Embeddable,
Governed Datasets
Curated
Data
Business
Apps
Self-Service
Analytics
Predictive
Models
Machine Learning Orchestration - Solution
Pentaho streamlines the entire machine learning workflow and enable teams of data
scientists, engineers and analysts to train, tune, test and deploy predictive models
1 1 2
3
4Customer
Provisioning
Billing
Location
Web
Social
Media
Network
Notebook IntegrationsData Prep
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Intelligent DataOps – Pentaho Foundation
End-to-end Analytics Platform
Big Data Accelerator
IoT Enabler
Machine Learning Orchestrator
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Solutions through 3 use cases
Enablement &
collaborationVolume Real time ML
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Case study
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Alarm and process collection
correlation and analytics
Heat exchange network
data collection, analytics, and data science
Robust Quality Estimator (RQE) system
data analytics and data science
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Many data
silos with
LOB
disconnect
Data volumes
growing –
100,000 data
sets / day
Manual
processes
and lack of
data maturity
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Improved
lead time
from 2 days
to 10 minutes
Delivered
data
science
enablement
Unified data
architecture
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Case study
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Started by UK Government 2015 to reduce
travel time between London, Leeds,
Newcastle and Edinburgh
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Hitachi Class 800: 20+ UK Suppliers
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Collect Data From Trains
Load Information into Data Lake
Perform Computations for
Predictive Maintenance
(Machine Learning)
Perform Computations
for Signal Analysis
(Engineering)
Perform Computations
for Operations (PCS –
Passenger Count
System)
3.6 Millions Data Points /s(Hundreds of Trillions in 2years
!!)3000+ Sensors
Provide multi-tenant visualisation
20+ hardware suppliers
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Solutions
• Pentaho Big data Platform
• End to end solution
• No code data prep
• ML automation
• Governance of data
Benefits
• Train Presence rate is optimized
• Better service for customers
• 1million+ £ in savings on maintenance
Pentaho Data
Integration
Pentaho Business
Analytics
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Case study
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Fuel usage is a major cost saving opportunity
In many cases
fuel cost
accounts for
30% of revenue
in a shipping
company
Fuel prices are
steadily rising,
as are tariffs on
high emissions
Customers are
increasingly
conscious of
reducing
environmental
impact
OPTIMISING FUEL USAGE CAN SAVE YOU MONEY, IMPROVE YOUR OPERATIONS,
AND GROW YOUR CUSTOMER BASE
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
AI enabled support to allow a captain take decisions based on
facts to optimize fuel consumption
Route planning optimization is
dependent on varying a ship's
speed in different parts of a route.
Since the external conditions
(wind, current and depth) vary,
fuel consumption cannot be
maintained at a minimum, if a
constant speed and direction is
kept throughout the route.
Therefore we have to find the
speed distribution and optimal
direction that minimizes the total
fuel consumption within the
constraint of keeping the
scheduled arrival time.
DESTINATION
The background..
Captain, reduce speed by 1 knot and
change direction with 1 degree east.
That will save you yy kg fuel.
AI ASSISTANT TO IMPROVE FUEL CONSUMPTION
OK, assistant. We saved xx%
on the last journey we made so I
trust you.
A captain assisted by a
trained AI to make the best
possible decisions in order
to improve fuel efficiency.
Scaling of the best
knowledge and make it
available to all captains
through a highly trained AI
assistant
The vision..
START
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
High level Machine Learning approach for PoC.
Feature selection and parameter learning
Data extraction Feature selection Relevant featuresLearn the
relationshipValidate
▪ water depth
▪ weather changes
▪ number of engaged engines
▪ water current, direction and
speed
▪ winds and waves, direction and
speed
▪ speed relative to ground
▪ speed through water
▪ Aerodynamical hull design
▪ Hull cleaning interval
▪ trim of the vessel
▪ pitch angle
▪ load balancing
▪ propeller RPM, torque
▪ bearing or routes
▪ fuel consumption of past trips
▪ Propulsion power
▪ …
▪ …
Input (history)Neural network
or any other ML
algorithm
▪ weather changes
▪ water current, direction and
speed
▪ winds and waves, direction
and speed
▪ speed relative to ground
▪ speed through water
▪ Hull cleaning interval
▪ trim of the vessel
▪ fuel consumption of past
trips
▪ bearing or routes
▪ Propulsion power
▪ …
▪ …
OutputInput
▪ weather changes
▪ water current, direction and speed
▪ winds and waves, direction and speed
▪ speed relative to ground
▪ Hull cleaning interval
▪ trim of the vessel
▪ fuel consumption of past trips
▪ bearing or routes
▪ …
▪ 𝒇𝒇𝒖𝒆𝒍𝒄𝒐𝒔𝒕(𝑠𝑝𝑒𝑒𝑑, 𝑡𝑟𝑖𝑚, 𝑐𝑢𝑟𝑟𝑒𝑛𝑡,𝑤𝑖𝑛𝑑. . )
▪ 𝒗𝒔𝒑𝒆𝒆𝒅 𝑡𝑟𝑖𝑚, 𝑐𝑢𝑟𝑟𝑒𝑛𝑡,𝑤𝑖𝑛𝑑, 𝑏𝑒𝑎𝑟𝑖𝑛𝑔, …
▪ 𝒓𝒕𝒓𝒊𝒎 𝑠𝑝𝑒𝑒𝑑, 𝑐𝑢𝑟𝑟𝑒𝑛𝑡,𝑤𝑖𝑛𝑑, 𝑏𝑒𝑎𝑟𝑖𝑛𝑔 …▪ 𝒙𝒓𝒐𝒖𝒕𝒆(𝑠𝑝𝑒𝑒𝑑, 𝑡𝑟𝑖𝑚, 𝑐𝑢𝑟𝑟𝑒𝑛𝑡,𝑤𝑖𝑛𝑑, … )▪ …
▪ speed
▪ bearing or routes
▪ Trim
▪ estimated fuel
consumption
▪ …
▪ …
▪ …
Input (present)
Output
Learnt
functions
Neural network
or any other ML
algorithm
© Hitachi Vantara Corporation 2019. All Rights Reserved.© Hitachi Vantara Corporation 2019. All Rights Reserved.
Fuel Optimization Dashboard
© Hitachi Vantara Corporation 2018. All Rights Reserved© Hitachi Vantara Corporation 2019. All Rights Reserved.
Thank You
Pol Huyzentruyt
Hitachi Vantara Belux
Senior Solutions Consultant
© Hitachi Vantara Corporation 2018. All Rights Reserved