Hortonworks Data In Motion Series Part 3 - HDF Ambari

Harnessing Data-in-Motion with Hortonworks DataFlow

HDF 2.0: Apache Ambari Integration

Ali BajwaPrincipal Partner Solutions Engineer

Yolanda DavisSenior Software Engineer

Recap

HDF 2.0: Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm, Apache Ambari

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Hortonworks DataFlow Manages Data-in-MotionCore

InfrastructureSources

Constrained High-latency Localized context

Hybrid – cloud / on-premises Low-latency Global context

RegionalInfrastructure

Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm are trademarks of the Apache Software Foundation

https://nifi.apache.org/minifi/

https://nifi.apache.org/

https://nifi.apache.org/


What is Apache Ambari?

Health Checks, Alerts Stacks, Views

Lifecycle controls, Rolling Restarts, Decommission/Re-

commission

Host Groups, Versioning, Compare, Revert,

Recommendations, Security Setup

Install Wizard (UI),Blueprints (API)

ConfigManagement

ExtensibilityMonitoring

ServiceManagement

ClusterProvisioning


AgendaApache Ambari Integration in HDF 2.0• Deployment• Configuration• Debugging• Monitoring

Ease of Deployment


Ease of Deployment

Deploy Apache NiFi through:– Apache Ambari install wizard or – Operationalize via blueprints automation

Choose which nodes Apache NiFi can be installed on

So either:– Choose Apache NiFi hosts at time of

cluster install


Ease of Deployment– Or add Apache NiFi to existing host (post-cluster install) and then start it


Ease of Deployment–Can configure which user/group Apache NiFi runs as (when cluster installed or when Apache NiFi is added to existing cluster)


Ease of Deployment Can remove Apache NiFi service from Apache Ambari

– Note: this does not remove bits Can stop/start/configure Apache NiFi across cluster via:

– Apache Ambari UI or – Apache Ambari REST API


Ease of Deployment Can stop/start Apache NiFi on individual hosts via Apache Ambari UI and also via

REST API


Ease of Deployment

Quicklink for easy access to Apache NiFi UI – URL dynamically determined based on:

• whether SSL enabled and • which ports user configured

Demo #1: Deploy HDF 2.0


Demo #1: Deploy HDF 2.0

•Deploy HDF cluster using Apache Ambari Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html

–Install Apache Ambari–Install HDF mpack–Open Apache Ambari WebUI–Run through Install Wizard

http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html


Ease of Configuration



Configure once via Apache Ambari (instead of repeating config on each node) Support for most Apache NiFi config files (e.g. nifi.properties, bootstrap.conf etc) Assistance for admin during Apache NiFi configuration:

– Help text displayed on hover with property descriptions– Checkboxes instead of true/false values– User friendly labels and default values– ‘Computed’ values can be automatically handled (e.g. node address)

Get benefit of Apache Ambari config features:– Config history:

• diff between versions, revert to older version etc– Update configs via Apache Ambari REST API




Ease of Configuration•Manage host specific configs using Config groups feature

–e.g. to bump memory for Apache NiFi on one of the nodes, click Override button:


Ease of Configuration ...create new config group and add desired host to it



‘Common’ configs grouped together and exposed in first config section (‘Advanced nifi-ambari-config’) to allow configuration of:– Ports (nonSSL, SSL, protocol)– Initial and max memory (Xms, Xmx)– Repo default dir locations (provenance, content, db, flow file)– ‘Internal’ dir location - contains files NiFi will write to

• ‘conf’ subdir for flow/tar.gz, authorizations.xml• ‘state’ subdir for internal state• Can change subdir names by prefixing the desired subdir name with

‘{nifi_internal_dir}/’– Sensitive props key (used to encrypt sensitive property values)– Zookeeper znode for nifi





nifi.properties exposed under ‘Advanced nifi-properties’ as key/value pairs with helptext– values replaced by Apache Ambari shown surrounded by double braces e.g.{{ }} but

can be overridden by end user– Properties can be updated or added via ‘Custom nifi-properties’ and will get written

to all nodes– ‘Computed’ values handled:

• ‘Node address’ fields replaced with each host's own FQDN




Other property based config files exposed as jinja templates– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden by

end user– Properties can be added/updated and will get written to all nodes



Ease of Configuration Other xml based config files also exposed as jinja templates

– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden – Elements can be updated/added and will get written to all nodes

– Config files written out with either 0400 or 0600 perm

• Why? • Some property files contain plaintext passwords

Ease of Debugging


Ease of Debugging Logsearch integration for ease of visualizing/debugging Apache NiFi logs w/o connecting via SSH to

hosts (e.g nifi_app.log, nifi_user.log, nifi_bootstrap.log) By default, monitors FATAL,ERROR,WARN messages (for all HDF services)


Ease of Debugging Can view/drill into errors at component level or host level Can filter errors based on severity (fatal, error, warn, info, debug, trace) Can exclude ‘noisy’ messages to find the needle in the haystack

Ease of Monitoring


Ease of Monitoring Apache NiFi Service check: Used to ensure that Apache NiFi UI is up. Can also be

invoked via REST API for automation


Ease of Monitoring Apache NiFi Alert: Host level Apache Ambari alert when Apache NiFi process

goes down– Can temporarily be disabled by turning on maintenance mode


Ease of Monitoring Alerts tab allows:

– Disabling and Configuring alerts (e.g. change interval)


Ease of Monitoring Can setup email or

SNMP notifications to admins


Ease of Monitoring Apache Ambari Metrics (AMS) integration

– Service autocreates Apache Ambari reporting task, pointing to AMS collector host/port (autodetected)


Ease of MonitoringThis is achieved by setting up an initial flow.xml (configurable via Amari)...

Tip: can auto-deploy any Apache NiFi flow by pasting its flow.xml here


Ease of Monitoring...and passing arguments via bootstrap.conf Ensures Apache NiFi autodetects updated location of AMS collector (in case it’s moved)


Ease of Monitoring Results in AMS dashboard

for Apache NiFi metrics, such as:– Flowfiles sent/received– MBs read/written– JVM usage/thread counts


Ease of Monitoring

Dashboard widgets can:– be drilled into to see

results from last 1,2,4 hours, day, week etc

– export metrics data as csv or JSON


Ease of Monitoring Same metrics can be viewed in Grafana dashboard

– Access via quicklink under ‘Ambari metrics’ service– Allows users to easily create custom dashboards for each component


Summary - Apache NiFi in Apache Ambari

• HDF available via management pack (mpack) that defines required services, stack definitions etc

• Wizard Driven Setup• Provides access to Apache NiFi configurations, start/stop/restart• Has a Reporting Task for Apache Ambari Metrics• In Cluster Mode by Default• Leverages Zookeeper on HDF for State Management (not Embedded Zookeeper)• Some Stack Advisor configurations (Ranger integration, Apache NiFi Certificate

Authority)• Audit logging via Apache Ambari Infra (Internal Solr)

* You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is already installed (requires a dedicated instance)

Demo #2: Explore HDF 2.0


Demo #2: Explore HDF 2.0

Explore HDF 2.0– Apache NiFi configurations– Configuration groups– Metrics– Logsearch integration– Service checks– Alerts

Try it out!


Try it out!

Deploy HDF cluster using either:– a) Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html

– b) Blueprints: https://community.hortonworks.com/articles/56849/automate-deployment-of-hdf-20-clusters-using-ambar.html

* Reminder: You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is already installed (requires a dedicated instance)








https://community.hortonworks.com/articles/56849/automate-deployment-of-hdf-20-clusters-using-ambar.html




Questions? https://community.hortonworks.com/

Hortonworks Community Connection:Data Ingestion and Streaming

Date post:	16-Apr-2017
Category:	Technology
Upload:	hortonworks
View:	2,798 times
Download:	8 times

Hortonworks Data In Motion Series Part 3 - HDF Ambari

Technology