Date post: | 25-Jun-2015 |
Category: |
Education |
Upload: | bdpa-education-and-technology-foundation |
View: | 628 times |
Download: | 0 times |
Mayo Clinic Big Data Projects
Experience
Brian Brownlow
Big Data Professional
04/13/2023 2
What is Big Data?• A silver bullet that will solve all the worlds problems? NO
• An arrow in the IT quiver to help solve customer problems? YES
• Does anyone have large data problems? All sales transactions, log reviews, device output, text processing?
• How does you relational DB handle index creation or backup for 500,000,000,000 row tables?
• Popular things that are similar• Seti, many networked computers doing small pieces of work• Watson, many networked computers working together to
solve a problem• What’s one computer that beat a chess master? Kasparov –
Deep Blue (1996–1997), there are others…
• Big data has been around a long time
• Why now? Bigger, cheaper, faster processing, memory, networking and disk
04/13/2023 3
Mayo Big Data Elements
• Patient Information• Appointments• Labs• Images• Genome• Appointment Check-in/Check-out• Report text• Vitals• Device reporting, e.g. Holter Monitor• Many more, it keeps growing…
04/13/2023 4
Mayo Big Data Elements Potentially Affecting Patient Care
• ALL OF THEM!
• The more we know about a patient the better we can build tools and models to help the care team improve patient care and help the business manage to reimbursement.
04/13/2023 5
Mayo Big Data Initial Evaluations
• Hortonworks HDP on a Virtual Machine on my laptop• HDP 1.3.2, 2.1 on Oracle VM• HDP 1.3.2, 2.1 on VMWare
• What can HDP do?• Pig, Hive, Hbase, HDFS, Ambari,
Hue, MapReduce, FLUME, Storm, ElasticSearch, Sqoop…
04/13/2023 6
Mayo Big Data Presentations to Leadership
• What is “BIG DATA”? What is Hadoop?
• What are “BIG DATA” capabilities?
• Here is one way you can answer your customer queries about big data!
• Many people want to have a “BIG DATA” story
• Proved out at Mayo by some initial proof of concept projects
• Genomics on Cloudera (early work)• HDP on Oracle VM (my project)• Multi node DEV environment on HDP 1.3.2 running Centos on
XenCenter and an outside edge node
• Helped by media hype.
04/13/2023 7
The Virtual Machine!
• Show it.
04/13/2023 8
Mayo Big Data DEV
04/13/2023 9
Big Data DEV Setup
• Lots of help on the web, Hortonworks website, other websites
• Using the latest version of CentOS: 6.5 (x64)
• Exported VM to CentOS6.5_Hadoop1.32_SSD3.ova
• Installed as a VM from Oracle Virtual Box on Citrix XenCenter
• Installed or Updated latest packages for yum, rpm, wget, curl, scp, pdsh, …
• Downloaded and generated local HDP repository /etc/ yum.repo.d (Note: 3 versions HDP hadoop stacks – 1.3.2, 1.3.3, 2.0.6)
• Configured network (hosts, security, firewall…)
• Installed Ambari (v1.4.4.23) and embedded postgresql DB (v8.4.18)
• Installed Hadoop components from Ambari
04/13/2023 10
Big Data DEV Environment
• Was it Perfect? NO• Less stable than preferred due to enabled
updates• Lightly used• Checked daily• By the time of heavier we had our INT and
PROD environments so we didn’t need DEV
• Was It Good Enough? YES
04/13/2023 11
Mayo Big Data Platform RFP
• Sent out RFP, got demos based on a use case we submitted with the RFP
• IBM Big Insights• Cloudera Hadoop Distribution• TeraData/Hortonworks Hadoop Distribution
• Selected TeraData/Hortonworks on a TeraData hardware frame
• TDH (Teradata Hadoop is not a exact copy of HDP (Hortonworks Data Platform)
• TeraData brings appliance brings some good things to the table, Viewpoint, HCLI, …
04/13/2023 12
Big Data INT and PROD
• TDH INT in one cabinet, TDH PROD in the other, asked Teradata for a VM version
• Additional expansion space available in existing INT and PROD racks, want a big data project? Fund a new edge or data node!
• TeraData add-ons, RAID, Infiband, Viewpoint, HCLI
• TDH 1.3.2 not HDP 1.3.2, same source base but minor differences to support the TeraData infrastructure
• Ideal: DEV=INT=PROD, hardware and software
04/13/2023 13
Master Prod 2
Master Prod 1
Edge Prod 2
Data Prod 6
Data Prod 5
Data Prod 4
Data Prod 3
Data Prod 2
Edge Prod 1
Data Prod 1
Primary SM Enet Switch
System VMS
Network-0 InfiniBand Switch
KVM
Cabling Slot
Network-1 InfiniBand Switch
Space for
Additional Nodes
Secondary SM Enet Switch
Master Test 2
Master Test 1
Edge Test 2
Data Test 6
Data Test 5
Data Test 4
Data Test 3
Data Test 2
Edge Test 1
Data Test 1
Primary SM Enet Switch
Cabinet VMS
Space for
Additional Nodes
Secondary SM Enet Switch
Viewpoint TMS
• 20 Hadoop nodes total – 10 per cabinet
• 2 Hadoop clusters, one per cabinet:
• Prod: 2 Master, 2 Edge, 1 Viewpoint TMS, 6 Data nodes (can add up to 7 more Edge and/or Data nodes in-cabinet, plus add additional cabinets to the cluster)
• Integration Test: 2 Master, 2 Edge, 6 Data nodes (can add up to 8 more Edge and/or Data nodes in-cabinet, plus add additional cabinets to the cluster)
• Raw user data capacity per cluster: 57+ TB
• Includes HDFS 3x replication & work space
• Does NOT include any compression!
• Example: at 2x compression, user data space per cluster is 114+ TB
• Power: 3 phase; 2 x 60 amps per cabinet; bottom egress
• HDP 1.3.2; Storm, Elasticsearch, and WebSphere MQ to be installed on appliance by project team
• Teradata Managed Server (TMS) for Viewpoint
TDH INT and PROD
04/13/2023 14
Big Data Project Setup
• Agile development – 2 week sprints, daily scrums
• Extreme Programming
• Java Development Environment tool tree• SVN (Subversion)• Jenkins• Maven• Eclipse – Kepler
• Open Source Components• Storm • Flume• Elastic Search (Marvel)• NLP - cTAKES
• Acquired training for all components as needed, e.g. Storm, Flume, Elastic Search, SVN, Drools
• Used in DEV, INT and PROD environments
• Consulting engagements
04/13/2023 15
DEV Team
• The team• Executive support• Project manager • Senior Technical staff member• 4 very experienced Programmers• Very motivated, flexible, hearts of teachers and
learners
• Agile and Extreme programming relatively new to Mayo IT
• Parts of the tool tree were also relatively new to Mayo IT
04/13/2023 16
Part 1
• Verify the development tool tree
• Verify the development process
• Verify the open source components
• Define first use cases
• Start and manage the project backlog list
04/13/2023 17
Part 1 Projects
• Natural Language Processing• Lets get more value from unstructured text!
• Standard big data use cases• Exploration
• Log exploration• Search• …
• Data lake• Cohort identification• …
04/13/2023 18
Part 1 Pig, Hive
PIGA = LOAD 'default.bnb_test_from_file' USING org.apache.hcatalog.pig.HCatLoader();
DUMP A;
Hive'SELECT * FROM default.bnb_test_from_file limit 2'
04/13/2023 19
Part 1
• In production!
• Well received
• Met expectations for the development process and schedule
• Lots of people lined up now to use the environment!
04/13/2023 20
Part 2
• More NLP work• Get more source data from more sources• Explore via Drools, ElasticSearch, MapReduce
• Many more lined up• Security – log examination• Clinical Trials cohort discovery• Genomics/Phenomics• Molecular biology• Protein studies• …
04/13/2023 21
Conclusion
• Big Data via Hadoop is a relivent choice in certain problem spaces
• Open source can provide valuable tools for our customers
• Questions?