14
FalĐoŶ Useƌ IŶteƌfaĐe: TeĐhŶiĐal Pƌeǀieǁ
Fal o Use I te fa e: Te h i al P e ie
Wel o e to Ho to o ks Fal o Use I te fa e UI Te h i al P e ie . This Te h i al P e ie p o ides ea l a ess to apa ilities e a e u e tl o ki g o fo a futu e elease. The Te h i al P e ie lets ou e plo e these featu es a d p o ide a oppo tu it to i o po ate use feed a k du i g the de elop e t p o ess. These featu es a e o side ed u de de elop e t a d a e ot i te ded fo use i ou p odu tio s ste s a d a e ot suppo ted
Ho to o ks ut ou feed a k is g eatl app e iated. Ha e fu a d please se d feed a k o the Ho to o ks Co u it fo u s
http://ho to o ks. o / o u it /fo u s/fo u /fal o /
Fal o UI Te h i al P e ie Fal o Vie s e a les i h API fu tio alit i a i tuiti e a d st ea li ed e i te fa e to eate a d a age data feeds, p o ess, luste a d i o e tities a d thei i sta es. Additio all , i this
p e ie sea h a d li eage apa ilities ha e ee i teg ated. The pu pose of this do u e t is to gi e detailed i st u tio s o ho to i stall a d p e ie the e Fal o UI u i g o HDP . . . . Please ote that fo the GA elease, the Fal o UI ill e u i g o HDP . ‐ thus the e Fal o featu es i HDP . ill NOT o k ith this p e ie . Additio all , e a e aki g os eti ha ges to the look a d feel of the a ious s ee s. The flo a d featu es ill e ai the sa e ut e pe t fu the efi e e t i the GA elease. Boot “t ap I st u tio s : “et up a sa d o o ag a t ode ith Hadoop dist i utio . . . o late . The efe e e steps a e ased spe ifi all o . . . ‐ .
You a get the VM he e:
http://ho to o ks. o /p odu ts/ho to o ks‐sa d o /#i stall Ve if o figu atio a d setup fo Fal o as outli ed he e, ut do ot eate e tities. This ust e do e to ake su e all p e e uisites a e i pla e.
http://ho to o ks. o /hadoop‐tuto ial/defi i g‐p o essi g‐data‐e d‐e d‐data‐pipeli e‐apa he‐fal o Hi t: Get sa ple files ha e the ha d fo p e ie steps. F o ith the VM u this o a d. wget http://hortonassets.s3.amazonaws.com/tutorial/falcon/falconDemo.zip
I stall the e Fal o p e ie UI: “top Fal o
● As use fal o , please do the follo i g: bash# cd /usr/hdp/<version>/falcon bash# ./bin/falconstop
Ba kup E isti g Ve sio of Fal o
● As use oot, please do the follo i g: ## backup falcon bash# cd /usr/hdp/<version>/falcon/server/webapp bash# mkdir /tmp/falconbackup bash# cp r * /tmp/falconbackup/
I stall Fal o UI P e ie Bits
● Do load the fal o . a f o : http://publicrepo1.hortonworks.com/HDPLABS/Projects/Falcon/falcon.war to a te po a lo atio su h as: /te p/fal o _UI bash# rm rf * bash# cp /temp/falcon_UI/falcon.war . bash# mkdir /usr/hdp/<version>/falcon/server/webapp/falcon bash# cd falcon bash# jar xf ../falcon.war bash# cd .. bash# chown R falcon:falcon . bash# chmod R 755 .
“etup Di e to Paths fo Cluste C eatio
bash# hadoop fs mkdir /apps/falcon/primaryCluster bash# hadoop fs mkdir /apps/falcon/backupCluster bash# hadoop fs mkdir /apps/falcon/primaryCluster/staging bash# hadoop fs mkdir /apps/falcon/primaryCluster/working bash# hadoop fs mkdir /apps/falcon/backupCluster/staging bash# hadoop fs mkdir /apps/falcon/backupCluster/working bash# hadoop fs chmod R 777 /apps/falcon/* bash# hadoop fs chown R falcon /apps/falcon/* bash# hadoop fs chmod R 755 /apps/falcon/primaryCluster/working /apps/falcon/backupCluster/working
“ta t Fal o ith p e ie UI i stalled. As use fal o , please do the follo i g:
bash# cd /usr/hdp/<version>/falcon bash# ./bin/falconstart
C eate use a a i‐ a As use oot , please do the follo i g:
bash# useradd G users ambariqa As use hdfs , please do the follo i g:
bash# hadoop fs mkdir /user/ambariqa bash# hadoop fs chown ambariqa /user/ambariqa bash# hadoop fs chmod 755 /user/ambariqa
“ta t e plo i g the Fal o UI Te h P e ie Featu es
● A ess the Fal o UI ia poi t ou o se to http:// . . . : / Logi to the e UI ith use a e a a i‐ a a d pass o d as ad i . Cu e t e sio of Ch o e o Fi efo e o e ded. You a do the follo i g:
○ Add luste , feed a d p o ess e tities ○ “ hedule, suspe d, edit a d delete feed, p o ess e tities ○ “ea h feed a d p o ess e tities ○ “et up a i o jo
Add Cluste , Feed, a d P o ess E tities Buildi g o the e isti g Fal o Tuto ial alo g ith the Fal o UI Te h i al P e ie , ou a le e age the apa ilities p o ided the UI to setup the luste , feed, a d p o ess e tities athe tha o k di e tl ith the u de l i g spe ifi atio files. Cluste Spe ifi atio Ea h luste ust ha e o e E tit spe ifi atio pe luste . This a e a o plished i t o a s. The fi st is di e tl editi g the l file as outli ed i the tuto ial. The se o d is th ough the e i te fa e p e ie . Use the sa e i fo atio i the XML file fo the e fo to eate e tities. “ee elo fo a sa ple luste XML spe ifi atio file. Do ot eate the XML file sho ut use the e fo to eate a d su it.
Cli k o the eate e tit utto fo luste :
Feed Spe ifi atio E tit A feed a.k.a dataset sig ifies the lo atio of data a d its asso iated epli atio poli a d late a i al ut‐off ti e. “ee elo fo a sa ple feed a.k.a dataset spe ifi atio file, ut as i the luste e tit eatio , use the e fo to eate o e.
Cli k the Feed e tit eatio utto :
Fill i the e fo as outli ed i the XML file.
Hi t : You a use the edit utto to ut a d paste a of the fields a d go a k to the e fo .
No let’s su it the sou e of the a e ail feed. This feed sig ifies the a e ails that a e ei g do loaded i to the Hadoop luste . These e ails ill e used the e ail lea si g p o ess. We a e o ead to defi e the feed e tit hi h ill ha dle the e d of the pipeli e to sto e the lea sed e ail. This feed sig ifies the e ails p odu ed the lea se e ail p o ess. It also takes a e of epli ati g the lea sed e ail dataset to the a kup luste Vi gi ia luste .
P o ess Spe ifi atio E tit A p o ess defi es o kflo o figu atio . A o kflo is a di e ted a li g aph DAG hi h defi es
the jo fo the o kflo e gi e. A p o ess defi itio defi es the o figu atio s e ui ed to u the
o kflo jo . Fo e a ple, p o ess defi es the f e ue at hi h the o kflo should u , the
luste s o hi h the o kflo should u , the i puts a d outputs fo the o kflo , ho the
o kflo failu es should e ha dled, ho the late i puts should e ha dled a d so o .
He e is a e a ple of hat a p o ess spe ifi atio looks like i l. Please eate the p o ess usi g
the e fo .
Cli k the P o ess e tit eatio utto .
Fill i the e fo as i p e iousl do e fo othe e tities.
The i gest p o ess is espo si le fo alli g the Oozie o kflo that do loads the a e ails f o the e i to the p i a Hadoop luste u de the lo atio spe ified i the a E ailFeed. l It also takes a e of ha dli g late data a i als. The lea se p o ess is espo si le fo alli g the pig s ipt that lea s the a e ails a d p odu es the lea e ails that a e the epli ated to the a kup Hadoop luste
Ma age “ hedule, Resu e, Pause the Fal o e tities To a age a jo , sea h fo the a e of the feed o p o ess a d sele t the app op iate utto . Fo the tuto ial sele t the app op iate e tit a d sele t s hedule.
Mi o E tit Additio all , ou a e plo e the setup of a Mi o , hi h is used to epli ate jo fo HDFS o Hi e DR
hole DB epli atio ‐‐ This ill ha e li ited to HDFS as the Hi e DR featu es e ui e Fal o .6. is eeded HDP . .
“et up i o jo a a i‐ a
● Do load o kflo . l f o ● … to /t p/
As use hdfs , do the follo i g.
bash# hadoop fs mkdir /apps/datamirroring/ bash# hadoop fs mkdir /apps/datamirroring/workflows bash# hadoop fs copyFromLocal /tmp/hdfsreplicationworkflow.xml /apps/datamirroring/workflows/ bash# hadoop fs chown R ambariqa /apps/datamirroring/
No , go to the UI a d li k o o Mi o . Whe ou su it i o jo , ake su e oth the sou e a d ta get paths e ist a d a e ead/ ita le use a a i‐ a
Tha ks fo taki g ou p e ie fo a test d i e. Please di e t ou feed a k to: http://ho to o ks. o / o u it /fo u s/fo u /fal o /
