Spark Advanced DevOps

7/23/2019 Spark Advanced DevOps

http://slidepdf.com/reader/full/spark-advanced-devops 1/219

!"#$%&

'!#'()"! )+'&&

,-./0 12345 &6-.7 &899:; "-<; 1234

0;;65==<>:?@<0-.@AB@;=?-;-C.:/7<

DDDA>:B7@?:BA/E9=:B=C>8@6>-<;:/



9-7:BF C:F ?-;- <:96>@

!-;-C.:/7< )>E8?5“A 8B:G:@? 6>-;GE.9 GE. C

– G.E9 "I+ ;E "J6>E.-;:EB '?L-B/@? 'B->M;:/< -B?

•

NE8B?@? :B >-;@ 123O• CM ;0@ /.@-;E.< EG '6-/0@ &6-.7

• $.:F:B-> ;@-9 G.E9 P) [email protected]@>@M ',%+-C

• Q-:<@? RST ,:>>:EB :B 1 .E8B?<

• U42 @96>EM@@<

• We’re hiring!

• +@L@> 1=O <866E.; 6-.;B@.<0:6< D:;0

• )>[email protected]

• VE.;EBDE.7<

• ,-6Q

• !-;-&;-J

W0;;65==?-;-C.:/7<ADE.7-C>@A/E9 X

http://databricks.workable.com/

http://databricks.workable.com/



I0@ !-;-C.:/7< ;@-9 /EB;.:C8;@? 9E.@ ;0-B !"# EG ;0@ /E?@ -??@? ;E &6-.



'Y"(!'

•

V:<;E.M EG &6-.7

• Q!! G8B?-9@B;-><

• &6-.7 Q8B;:9@ './0:;@/;8.@ZB;@F.-;:EB D:;0 Q@<E8./@ ,-B-F@.<W&;-B?->EB@K ['Q(X

• YPZ<

•

+-C5 !@L$6< 323

H@GE.@ +8B/0

•

,@9E.M -B? %@.<:<;@B/@

• \EC< ]^ &;-F@< ]^ I-<7<

• H.E-?/-<; #-.:-C>@< -B?'//898>-;E.<

• %M&6-.7

• !@L$6< 321

• &08GG>@

• &6-.7 &;.@-9:BF

'G;@. +8B/0



&E9@ <>:?@< D:>> C@ <7:66@?

%>@-<@ 7@@6 _`' >ED ?8.:BF />-<<

W469 – 45O269 GE. _`' D:;0 :B<;.8/;E.X

1 -BEBM9E8< <8.L@M<5 %.@ -B? %E<; />-<<

+8B/05 BEEB – 369

1 C.@-7< WC@GE.@ >8B/0 -B? -G;@. >8B/0X



a

• ',%+-C 6.Eb@/; D-< >-8B/0@? :B \-B 1233K c M@-. 6>-BB@? ?8.-;:EB

• %@.<EBB@>5 Uc4 <;8?@B;<K 6E<;?E/<K G-/8>;M ̀ <;-GG

• N8B?:BF G.E9 [email protected]@B;=ZB?8<;.M 6-.;B@.<0:6K (&N 'D-.?K !-.6-K !E"K12d /E96-B:@<

• ).@-;@? H!'&K ,@<E<K &('%A P6/E9:BF 6.Eb@/;<5 &8//:B/; ̀ #@>EJA

“Unknown to most of the world, the University of California, Berkeley’s ',%+0-< ->.@-?M >@G; -B :B?@>:C>@ 9-.7 EB ;0@ DE.>? EG :BGE.9-;:EB ;@/0BE>EFMK -even the web. But we haven’t yet experienced the full impact of thegroup[…] Not even close”

] !@..:/7 V-..:<K Y:F-$9K '8F 123S

'>FE.:;09<,-/0:B@<%@E6>@

https://gigaom.com/2014/08/02/the-lab-that-created-spark-wants-to-speed-up-everything-including-cures-for-cancer/





&/0@?8>:BF ,EB:;E.:BF !:<;



!"#$%&'()

!*+

,#%-./

9'8"#':5"'2(8;< =90< 09>< ?%-@< 9!A

B%C.D2(

?+E':

9%"%6#%&$8 7>4





%&'&()* +),-. /(0-&112'3

/(&3&*

4(&5&*

657)*)%()7.8)9

%2()7.

4(2**:&;

<=<,0(5

<7&-2)*2;&> <?1,&51

W:;@.-;:L@K :B;@.-/;:L@K ,+K <;.@-9:BFK F.-60K &_+K @;/X%&'&

W122S – 123OX

W122T – 1234eX

C).0D,



ED3 FGGH

AAA:B \8B@ 123O





FGH –





\8B@ 1232

0;;65==DDDA/<[email protected]@>@MA@?8=U9-;@:=6-6@.<=1232=

“The 9-:B -C<;.-/;:EB :B &6-.7 :< ;0-; ;.:C8;@? ?-;-<@; WQ!!XK D0:/0 .@6.@<@/E>>@/;:EB EG ECb@/;< 6-.;:;:EB@? -/.E9-/0:B@< ;0-; /-B C@ .@C8:>; :G - 6-.;

P<@.< /-B @J6>:/:;>M /-/0@ -B Q!! :B 9-/0:B@< -B? .@8<@ :; :B 98>;:6>@ ,-66-.->>@> [email protected];:EB<A

Q!!< -/0:@L@ G-8>; ;E>@.-B/@ ;0.E8F0>:B@-F@5 :G - 6-.;:;:EB EG -B Q!! :< >E<;@BE8F0 :BGE.9-;:EB -CE8; 0ED :; D-<E;0@. Q!!< ;E C@ -C>@ ;E .@C8:>? b8<; ;



'6.:> 1231

0;;65==DDDA/<[email protected]@>@MA@?8=U9-;@:=6-6@.<=1231

“We 6.@<@B; Q@<:>:@B; !:<;.:C8;@? !-;-?:<;.:C8;@? [email protected] -C<;.-/;:EB ;0-; >6.EF.-99@.< [email protected] :B][email protected] /E9>-.F@ />8<;@.< :B - G-8>;];E>@.-B; 9-BB

Q!!< -.@ 9E;:L-;@? CM ;DE ;M6@< EG -;0-; /8..@B; /E968;:BF [email protected]< 0:B@GG:/:@B;>M5 :;@.-;:L@ ->FE.:;09< -B? :9:B:BF ;EE><A

ZB CE;0 /-<@<K 7@@6:BF ?-;- :B 9@[email protected]/@ CM -B E.?@. EG 9-FB:;8

“Best %-6@. 'D-.? -B? VEBE.-C>@ ,@B;:EB GE. )

] ):;@? Og1 ;:9@<h



!"#$$%&'$#()*+&%,$%-$&%,.*1#($%&/2*3%$!%4$*+56*+5:6$;<=#6>5"/-%+56>

] 1 &;.@-9:BF %[email protected]<X 0-L@ C@@B /:;@?



)@(A$4 B 6%" D#E%A56$%4$/)&%):($) B )@(A$4*)@(/7-FGFA! H IJKL 8%58(%70

6,.%) B &%):($)*.,8/(,.M>,

Seemlessly mix SQL queries with Spark programs.

)E9:BF <EEBh

!"#$$ &' ()&$#*+', #- .+' )(/01#-23''4* 506 789:;< =>?@A



3&,8O B P&,8O/E%&$#+%)Q %.%)),3%) B )8,&9*$%4$I#(%3&,8OS B 3&,8O*T5#6U%&$#+

/#>Q E%&$%4Q .)30 BW **X

0;;6<5==-96>-CA/<[email protected]@>@MA@?8=D/EB;@B;=86>E-?<[email protected]:;0fGEB;<A6?G

https://amplab.cs.berkeley.edu/wp-content/uploads/2013/05/grades-graphx_with_fonts.pdf







0;;6<5==DDDA/<[email protected]@>@MA@?8=U<- [email protected]<M<3OA6?G

https://www.cs.berkeley.edu/~sameerag/blinkdb_eurosys13.pdf





0;;65==<0E6AE.@:>>MA/E9=6.E?8/;=2

@HEE75 ROOAgg%.:B;5 ROgAgg

%!NK @%8

&0:66:BF

0;;65==DDDA-9-jEBA/[email protected]:BFN-<;]!-;-]'B->M<:<=?6=3SSgO4ic1S

RO2 a '9-jEB5

http://shop.oreilly.com/product/0636920028512.do


http://www.amazon.com/Learning-Spark-Lightning-Fast-Data-Analysis/dp/1449358624









0;;65==;:BM8.>A/E9=?

] 321 6-F@<

] !@L$6< <;M>@

] NE. /E96>@;@ C

] ZB/>8?@<5

] &6-.7 &;.@-] !-BF@.< E

Y.E86HMk@Q@?8/@HM

http://tinyurl.com/dsesparklab





0;;65==;:BM8.>A/E9=/

] 32g 6-F@<

] !@L$6< <;M>@

] NE. /E96>@;@

] ZB/>8?@<5] %M&6-.7 ] &6-.7 &_+] &6-.7]<8C9

http://tinyurl.com/cdhsparklab









W&/->- ` %M;0EB EB>MX



!.:L@. %.EF.-9

!"!""

#

!""

I

I

!"

!""

#

!""II

lE.7@. ,-/0:B@

lE.7@. ,-/0:B@

# #



'"$&<F

'"$&<O'"$&<N'"$&<K'"$&<L

'"$&<P

'"$&<Q'"$&<R'"$&<S'"$&<FG

'"$&<FF

'"$&<FO'"$&<FN'"$&<FK'"$&<FL

'"$&<FP

'"$&<FQ'"$&<FR'"$&<FS'"$&<OG

'"$&<OF

'"$&<OO'"$&<ON'"$&<OK'"$&<OL

@99

!"!""

#

!""

!"!""

#

!""

!"!""

#

106' (B6.#.#0-* C



A##2#T "8T &8)FU%#(T "8T &8)OA##2#T "8T &8)F

@99 VW K -%#"'"'2(8

4(X2T "8T &8)RU%#(T "8T &8)O4(X2T "8T &8)R

A##2#T "8T &8)N4(X2T "8T &8)L4(X2T "8T &8)L

A##2#T "8T &8)KU%#(T "8T &8)SA##2#T "8T &8)F

'B Q!! /-B C@ /.@-;@? 1 D-M<5

] %-.->>@>:j@ - /E>>@/;:EB] Q@-? ?-;- G.E9 -B @J;@.B-> <E8./@ W&OK )mK V!N



Y Z,&,((%(#[% #6 Z<$O56"5&>)J\\ B )+*8,&,((%(#[%([“fish", “cats“, “dogs”])

RR Z,&,((%(#[% #6 -+,(,E,( "5&>)J\\B )+*8,&,((%(#[%/G#)$/71#)O7Q 7+,$)7Q 7>53)700

RR Z,&,((%(#[% #6 ],E,],E,J\\^-$&#63W "5&>)J\\ B )+*8,&,((%(#[%/_&&,<)*,)G#)$(“fish", “cats“, “dogs”)

] I-7@ -B/E>>@/;SparkC

9@;0E?

] (E; [email protected];E;M.@n8:.@9@9E.



Y J%,> , (5+,( $4$ 1#(% #6 Z<$O56(#6%)J\\ B )+*$%4$I#(%/7R8,$OR$5RJF_\LF*.>70

RR J%,> , (5+,( $4$ 1#(% #6 -+,(,E,( (#6%)J\\ B )+*$%4$I#(%/7R8,$OR$5RJF_\LF*.>70

RR J%,> , (5+,( $4$ 1#(% #6 ],E,],E,J\\^-$&#63W (#6%) B )+*$%4$I#(%/7R8,$OR$5RJF_\LF*.>70`

] I0@.@;E .@-)mK &O



A##2#T "8T &8)FU%#(T "8T &8)OA##2#T "8T &8)F

4(X2T "8T &8)RU%#(T "8T &8)O4(X2T "8T &8)R

A##2#T "8T &8)N4(X2T "8T &8)L4(X2T "8T &8)L

A##2#T "8T &8)KU%#(T "8T &8)SA##2#T "8T &8)F

A##2#T "8T &8)F

A##2#T "8T &8)F

A##2#T "8T &8)N A##2#T "8T &8)K

A##2#T "8T &8)F

IX'E"$#Y Z



IC2%E$8C$Y O Z

A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F

A##2#T "8T &8)K

A##2#T "8T &8)F

CE$%($1@99

A##2#T "8T &8)F

A##2#T "8T &8)F

A##2#T "8T &8)N A##2#T "8T &8)K

A##2#T "8T &8)F

IC2EE$C"Y Z

!.:L@.



IC2EE$C"Y Z

"J@/8;@ !'Yh

!.:L@.



IC2EE$C"Y Z

!.:L@.

E2)+'($8@99



IC2EE$C"Y Z

E2)+'($8@99

$##2#8@99

CE$%($1@99

IX'E"$#Y Z

IC2%E$8C$Y O Z

!.:L@.

A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F

A##2#T "8T &8)K

A##2#T "8T &8)F



IC2EE$C"Y Z

!.:L@.

E2)+'($8@99

$##2#8@99

CE$%($1@99

1%"%

IX'E"$#Y Z

IC2%E$8C$Y OT 8.5XXE$[ 6%E8$Z

>'-!"%



!.:L@.

E2)+'($8@99

$##2#8@99

CE$%($1@99



!.:L@.

1%"%



E2)+'($8@99

$##2#8@99

A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F

A##2#T "8T &8)K

A##2#T "8T &8)F

CE$%($1@99

IX'E"$#Y Z

A##2#T "8T &8)F

A##2#T "8T &8)F A##2#T "8T &8)F

$##2#IC2EE$C"Y Z

I8%\$B2=%88%(1#%Y Z

IC25("Y Z

L



E2)+'($8@99

$##2#8@99

A##2#T "8T &8)FA##2#T "8T &8)NA##2#T "8T &8)F

A##2#T "8T &8)K

A##2#T "8T &8)F

CE$%($1@99

IX'E"$#Y Z

A##2#T "8T &8)F

A##2#T "8T &8)F A##2#T "8T &8)F

$##2#IC2EE$C"Y Z

IC25("Y Z

I8%\$B2=%88%(1#%Y Z

L



><F E2)+'($8@99Y0%122-@99Z

><O ><N ><K

><F $##2#8@99YX'E"$#$1@99Z

><O ><N ><K

I-<7]3I-<7]1

I-<7]OI-<7

%-;0 o 0?G<5==A A A

G8B/ = _.contains(…)

<0E8>?)-/0@oG-><@

!-;-<@;]>@L@> L:@D5 %-.;:;:EB]>@L@> L:@D5



3X ).@-;@ <E9@ :B68; Q!!< G.E9 @J;@.B-> ?-;- E. 6-.->>@

/E>>@/;:EB :B ME8. ?.:L@. 6.EF.-9A

1X +-j:>M ;.-B<GE.9 ;0@9 ;E ?@G:B@ B@D Q!!< 8<:BF;.-B<GE.9-;:EB< >:7@ 1#($%&/0 E. .,8/0

OX '<7 &6-.7 ;E +,+O%/0 -BM :B;@.9@?:-;@ Q!!< ;0-; D:>> B

C@ .@8<@?A

SX +-8B/0 -/;:EB< <8/0 -< +5:6$/0 -B? +5((%+$/0 ;E 7:/6-.->>@> /E968;-;:EBK D0:/0 :< ;0@B E6;:9:j@? -B? @J@CM &6-.7A

W>-jMX



.,8/0 #6$%&)%+$#56/0 +,&$%)#56/0

1(,$L,8/0 >#)$#6+$/0 8#8%/0

1#($%&/0 3&5:8;<a%</0 +5,(%)+%/0

.,8Z,&$#$#56)/0 &%>:+%;<a%</0 &%8,&$#$#56/0

.,8Z,&$#$#56)=#$Ob6>%4/0 )5&$;<a%</0 8,&$#$#56;</0

),.8(%/0 T5#6/0 ***

:6#56/0 +53&5:8/0 ***

] ,E<; ;.-B<GE.9-;:EB< -.@ @>@9@B;]D:<@ W;0@M DE.7 EB EB@ @>@9@B; -; - ;:9@XK C;.8@ GE. ->> ;.-B<GE.9-;:EB<



&%>:+%/0 $,9%K&>%&%>/0

+5((%+$/0 ),E%_)!%4$I#(%/0

+5:6$/0 ),E%_)-%@:%6+%I#(%/0

1#&)$/0 ),E%_)KMT%+$I#(%/0

$,9%/0 +5:6$;<a%</0

$,9%-,.8(%/0 15&%,+O/0

),E%!5A,)),6>&,/0 ***



• #$%&&'!""

• ()*+,-,%!""

• .$'',%!""

• /$)-!""

• 01233*,%!""

• 45)&5!""

• /6+1&5!""

• "&27*,!""

• 8%79!""

• 8:&5!""

• 091,;$!""

• <,-+,=!""

• >%?,!""

• @$::$5%-$!""

• A,&!"" !(%)*'

• >:0'$-B !(+#







3X &@; EG 6-.;:;:EB< (“splits”)

1X +:<; EG ?@6@B?@B/:@< EB 6-.@B; Q!!<

OX N8B/;:EB ;E /E968;@ - 6-.;:;:EB F:L@B 6-.@B;<

SX $6;:EB-> 6.@G@..@? >E/-;:EB<

4X $6;:EB-> 6-.;:;:EB:BF :BGE GE. 7=L Q!!< W%-.;:;:[email protected]

:.21 -)7,D(&1 )** -D((&', <7)(O 07&(),20'1P

m

m

m

m

m



%-.;:;:EB< o EB@ 6@. V!N& C>E/7

!@6@B?@B/:@< o BEB@

)E968;@ W6-.;:;:EBX o .@-? /E..@<6EB?:BF C>E/7

6.@G@..@?+E/-;:EB< W6-.;X o V!N& C>E/7 >E/-;:EB

%-.;:;:EB@. o BEB@

m

m

m

m

m



%-.;:;:EB< o <-9@ -< 6-.@B; Q!!

Dependencies = “one];E]one” on parent

)E968;@ W6-.;:;:EBX o /E968;@ 6-.@B; -B? G:>;@. :;

6.@G@..@?+E/-;:EB< W6-.;X o BEB@ W-<7 6-.@B;X

%-.;:;:EB@. o BEB@

m

m

m

m

m



%-.;:;:EB< o $B@ 6@. .@?8/@ ;-<7

Dependencies = “shuffle” on each parent

)E968;@ W6-.;:;:EBX o .@-? -B? bE:B <08GG>@? ?-;-

6.@G@..@?+E/-;:EB< W6-.;X o BEB@

%-.;:;:EB@. o V-<0%-.;:;:[email protected]<7<X

m

m

m

m

m



E,( +,)),6>&,J\\ B )+*+,)),6>&,!,M(%(“9)”, “.<$,M(%”).select(“colc1”, “colc3”).where(“colc5 = ?”, “blue”)

k@M<6-/@ I-C>@

p&@.L@. <:?@ /E>89B` .ED <@>@/;:EB

WGE. ?@->:BF D:;0 D:?@ .ED<X



!"#$" "&' !(#$) *&'++ ,- (#**./0 ./ # 12*"34 1#**#/5$#6./(2"6*(+."6*.7'8

:M:6$:d#8cefcfc?gcShNij 5*' *(#$) –9*(#$)61#**#/5$#6./(2"6*(+."6*.7':;<<<

=%(+5.% $52222 22R 22R22 222 22222R R22

2k kR 2 kR 2 lR 22R m2RR222R *22Rk2Q2R2R R2Rk2k E%&)#56 f*n*e

R2R

')#63 -+,(, E%&)#56 S*ef*g /],E, D5$-85$/!L0 ohc;#$ -%&E%& ULQ ],E,e*p*f2?e0!<8% #6 %48&%))#56) $5 O,E% $O%. %E,(:,$%>*!<8% NO%(8 15& .5&% #615&.,$#56*A&%,$#63 -8,&9A56$%4$***A&%,$%> )8,&9 +56$%4$**-8,&9 +56$%4$ ,E,#(,M(% ,) )+*!<8% #6 %48&%))#56) $5 O,E% $O%. %E,(:,$%>*

!<8% NO%(8 15& .5&% #615&.,$#56*

)+,(,W

I0@ +,)),6>&,*#68:$*)8(#$*)#[% 6-.-9@;@. ?@G-8>;< ;E 322K222A I0:< :< ;0@ -66.EJ:9-;B89C@. EG 60M<:/-> .ED< :B - <:BF>@ &6-.7 6-.;:;:EBA ZG ME8 0-L@ .@->>M D:?@ .ED<W;0E8<-B?< EG /E>89B<XK ME8 9-M B@@? ;E >ED@. ;0:< L->8@A I0@ 0:F0@. ;0@ L->8@K ;0@G@D@. &6-.7 ;-<7< -.@ /.@-;@?A ZB/.@-<:BF ;0@ L->8@ ;EE 98/0 9-M >:9:; ;0@ 6-.->>@>:<9level.”

0;;6<5==F:;08CA/E9=?-;-<;-J=<6-.7]/-<<-B?.-]/EBB@/;E.



<7)(O BQ&-D

<7)(ORII0''&-,0

IS T)U) 4(

] $6@B &E8./@] Z96>@9@B;@? 9E<;>M :B &/->-] &/->- d \-L- '%Z<] !E@< -8;E9-;:/ ;M6@ /EBL@.<:EB<

0;;6<5==F:;08CA/E9=?-;-<;-J=<6-.7]/-<<-B?.-]/EBB@/;E.





“Simple ;0:BF<<0E8>? C@ <:96>@K

/E96>@J ;0:BF<<0E8>? C@ possible”

] '>-B k-M



!",$5





] +E/->

] &;-B?->EB@ &/0@?8>@.

] ['Q(

] ,@<E<

&;-;:/ %-.;:;:

!MB-9:/ %-.;:;:EB:BF

!"#$%&'(

http://hadoop.apache.org/




8&7C-$9B,-

"DCC

.

.! .

.! .

.! .

.!

.

.

.

.

..

!! !!

$&$&$&$&

8C

"D "DCC "DCC CC

D$;,D&%,

DD





I/@1J



$%&' !" ) *+,-.+

421O

/**0 12 :)1O

/**0 13

/**0 12

/**0 13

/**0 14

:)1O

:)1O

:)1O

:)1O

:)1O

:)1O

:)1O

:)1O

:)1O

:)1O

:)1O

6',&(')*

:.(&)>1

E$* 9&53 F 5,H 0'$-B@&53IJ

K:,+.$:+,-IL*&9$*M

K:,+Q''D$;,I“.6()-

K:,+IL:'$-BK,=,92+

E$* :9 F 5,H 0'$-B@&5+,=+I9&53J

S KT7)5T:'$-BU:1,** UU;$:+,- *&

S KT7)5T:'$-BU:27;)+ UU5$;, L.6()-:+Q'

UU;$:+,- *&9$*MNO

lE.7@. ,-/0:B@









#

*+,-.+

&&!V< 421O

#

&&!V< 421O

#

V< 421O

#

&&!V< 421O

!" !" !"

*+,-.+

!"!" !" !"



*+,-.+

&&!V< 421O &&!V< 421O V< 421O

#

&&!V< 421O

!"

#

!"

#

!"

#

!"

#

!"

#

!"

#

!"

0/Q!WXYZ!W>!X]D0CQD@>0[ M%,3$2*+[ NP ^ &3 H&-B,- )5:+$59,: +& -25 &5 ,$91 ;$91)5,

0/Q!WXYZ!W>!X@Z!>0[ M%,3$2*+[ Q\\P ^ &3 9&-,: +& $**&H 0'$-B $''*)9$+)&5: +& 2:, &5 +1, ;$91

0/Q!WXYZ!W>!X.>.Z!_[ M%,3$2*+[ CZCQ\ !Q. – N A`P C&+$* ;,;&-6 +& $**&H 0'$-B $''*)9$+)&5: +&/EBG=<6-.7]@BLA<0

0/Q!WX"Q>.ZDX.>.Z!_[ M%,3$2*+[ aNO .`P .,;&-6 +& $**&9$+, +& +1, 0'$-B ;$:+,- $5% H&-B,- %$

&;-B?->EB@ <@;;:BF<



&;-B?->EB@ <@;;:BF<

] '66< <8C9:;;@? D:>> .8B :B NZN$ 9E?@ CM ?@G-8>;

)8,&9*+5&%)*.,4N 9-J:989 -9E8B; EG )%P /E.@< ;E .@n8@<; GE. ;0@-66>:/-;:EB G.E9 -/.E<< ;0@ />8<;@.

)8,&9*%4%+:$5&*.%.5&<N ,@9E.M GE. @-/0 @J@/8;E.

























Q@<E8./@,-B-F@.

I*2&', YK 3





(E?@,-B-F@. (E?@,-B-F@.

I0',)2'&(

(E?@,-B-

E77 C)1,&(

1 O S 4

I0',)2'

c c

T T

i

Q@<E8./@,-B-F@.

I*2&', YK

I’m HA via

W00X&&7&(





(E?@,-B-F@. (E?@,-B-F@.

I0',)2'&(

(E?@,-B-

E77 C)1,&( I0',)2'E77 C)1,&(

I0',)2'&( I0',)2'

I*2&', YF<-.&>D*&(

E771 C)1,&(

Q@<E8./@,-B-F@.

I*2&', YK

!.:L@.





(E?@,-B-F@. (E?@,-B-F@.

I0',)2'&(

(E?@,-B-

E77 C)1,&(

BQ&-D,0(

/** :

I0',)2'

BQ&-D

/** :

Q@<E8./@,-B-F@.

I*2&', YK





(E?@,-B-F@. (E?@,-B-F@.

I0',)2'&(

(E?@,-B-

E77 C)1,&(

BQ&-D,0(

/** :

I0',)2'

BQ&-D

/** :

4(2U&(I0',)2'&(

BQ&-D,0(

/** :

] !E@< BE; <

['Q( <@;;:BF<



cc6:.c%4%+:$5&)N /EB;.E>< 0ED 9-BM @J@/8;E.< D:>> C@ ->>E/-;@?

cc%4%+:$5&c.%.5&<N Q', GE. @-/0 @J@/8;E.

cc%4%+:$5&c+5&%)N )%P /E.@< GE. @-/0 @J@/8;E.

)8,&9*><6,.#+_((5+,$#56*%6,M(%>

)8,&9*><6,.#+_((5+,$#56*.#6F4%+:$5&))8,&9*><6,.#+_((5+,$#56*.,4F4%+:$5&))8,&9*><6,.#+_((5+,$#56*):)$,#6%>-+O%>:(%&;,+9(53!#.%5:$ /r0)8,&9*><6,.#+_((5+,$#56*)+O%>:(%&;,+9(53!#.%5:$ /L0)8,&9*><6,.#+_((5+,$#56*%4%+:$5&b>(%!#.%5:$ /a0

!MB-9:/ '>>E/-;:EB5

0;;6<5==F:;08CA/E9=-6-/0@=<6-.7=C>EC=9-<;@.=/E.@=<./=9-:B=</->-=E.F=-6-/0@=<6-.7="J@/8;E.'>>E/-;:EB,[email protected]<

YARN resource manager UI: O$$8NRR^#8 ,>>&%))WNtftt

W(E -66< .8BB:BFX



W(E -66< .8BB:BFX



M,9OU2:,-b)'UNcUcUdOUef gPh :'$-BU:27;)+ UU9*$::

&-?K$'$91,K:'$-BK,=$;'*,:K0'$-B/) UU%,'*&6U;&%, 9*),5+ UU;$:+

T&'+T9*&2%,-$T'$-9,*:T@"#UaKOKNUNK9%1aKOKNK'cKNOTV$-:T:'$-BU

,=$;'*,:UNKNKcU9%1aKOKNU1$%&&'OKaKcU9%1aKOKNKV$- Nc

'66 .8BB:BF :B />:@B;







M,9OU2:,-b)'UNcUcUdOUef gPh :'$-BU:27;)+ UU9*$::

&-?K$'$91,K:'$-BK,=$;'*,:K0'$-B/) UU%,'*&6U;&%, 9*2:+,- UU;$:

6$-5 T&'+T9*&2%,-$T'$-9,*:T@"#UaKOKNUNK9%1aKOKNK'cKNOTV$-:T:'

,=$;'*,:UNKNKcU9%1aKOKNU1$%&&'OKaKcU9%1aKOKNKV$- Nc

'66 .8BB:BF :B />8<;@.



'66 .8BB:BF :B />8<;@.



'66 .8BB:BF :B />8<;@. 9E?@







<7)(O I&',()* C)1,&( Z.0 1,)(,1 BQ&-D,0(1[ :)1O1 (D' 2'

80-)* qBEB@r V89-B C@:BF "J@/8;E.

<,)'>)*0'& &;-B?->EB@ ,-<;@. lE.7@. \#, "J@/8;E.

\E]^ ['Q( '66 ,-<;@. (E?@ ,-B-F@. "J@/8;E.

C&101 ,@<E< ,-<;@. ,@<E< &>-L@ "J@/8;E.

)8,&9c):M.#$ 6.EL:?@< - 8B:GE.9 :B;@.G-/@ GE.<8C9:;;:BF bEC< -/.E<< ->> />8<;@. 9-B-F@.<



M#6R)8,&9c):M.#$ cc.,)$%& )8,&9NRRO5)$Npfppcc%4%+:$5&c.%.5&< ef3

.<2)+&#8$*8<

&E8





!"/**0 12

/**0 13

/**0 12

: :

: :: :

6',&(')*:.(&)>1

Recommended to use at most only 75% of a macGE. &6-.7

,:B:989 "J@/8;E. 0@-6 <:j@ <0E8>? C@ i YH

Max Executor heap size depends… maybe 40 YH

,@9E.M 8<-F@ :< F.@-;>M -GG@/;@? CM <;E.-F@ >@L@<@.:->:j-;:EB GE.9-;



_#<A

1$8$#'%E'_$1



J\\*+,+O%/0 BB J\\*8%&)#)$/LFLKJu2KrGu0

]^?

1$8$#'%E'_$1

most CPU-efficient option



8$#'%E'_$1



J\\*8%&)#)$/LFLKJu2KrGu2-FJ0

]^?

8$#'%E'_$1

_



*8%&)#)$/LFLKJu2_r\2\b-a0

]^?

1$8$#'%E'_$1

V< 42

_



*8%&)#)$/LFLKJu2_r\2\b-a2-FJ0

]^?

8$#'%E'_$1



*8%&)#)$/\b-a2KrGu0

]^?

1$8$#'%E'_$1

1$8$#'%E'_$1



J\\*8%&)#)$/LFLKJu2KrGu2S0

]^? 2( `21$ / ]^? 2( `

_



*8%&)#)$/LFLKJu2_r\2\b-a2S0

]^?

1$8$#'%E'_$1 1$8$#'%E'_$1



*8%&)#)$/KII2DF_Z0

]^?<F W 7--<F

8$#'%E'_$1

B%C.D2(

]^?<O W 7--<F



*:68%&)#)$/0

]^?



]^?

?

b



] ZG Q!! G:;< :B [email protected] /0EE<@ ,",$Q[f$(+[

] ZG BE;K 8<@ ,",$Q[f$(+[f&"Q D= G-<; <@.:->:j-;:EB >:C.-.M

] Don’t spill to disk unless functions that computed the datase-.@ [email protected] @J6@B<:L@ E. ;0@M G:>;@. - >-.F@ -9E8B; EG ?-;-AW.@/E968;:BF 9-M C@ -< G-<; -< .@-?:BF G.E9 ?:<7X

] P<@ .@6>:/-;@? <;E.-F@ >@L@>< <6-.:BF>M -B? EB>M :G ME8 D-B; G

G-8>; .@/[email protected] W9-MC@ ;E <@.L@ .@n8@<;< G.E9 - D@C -66X

Q@9@[email protected]



ZB;@.9@?:-;@ ?-;- :< -8;E9-;:/->>M 6@.<:<;@? ?8.:BF <08GG>@ [email protected];:EB<

Q@9@[email protected]

o



%M&6-.75 <;E.@? ECb@/;< D:>> ->D-M< C@ <@.:->:j@? D:;0 %:/7>@ >:C.-.MK <E :; ?BE; 9-;;@. D0@;0@. ME8 /0EE<@ - <@.:->:j@? >@L@>A

!@G-8>; ,@9E.M '>>E/-;:EB :B "J@/8;E. \#,

@ /)8,&9*)$5&,3%*.%.5&



MG#FG#

FG#

I)-.&> ]441

@1&( /(03()51!6'1B#-,'6A

<.DAA*& 5&50(?

8 3

Ibv !Db-

Q!! &;E.-F@5 D0@B ME8 /->> A6@.<:<;WX E. A/-/0@WXA &6-.7 D:>> >:9:; ;0@ -9E8B; EGmemory used when caching to a certain fraction of the JVM’s overall heap se

&6-.7 8<@< [email protected] GE.5



memory used when caching to a certain fraction of the JVM s overall heap, se)8,&9*)$5&,3%*.%.5&<I&,+$#56

&08GG>@ -B? -FF.@F-;:EB C8GG@.<5 l0@B [email protected]:BF <08GG>@ [email protected];:EB<K &6-.7 D:>>/.@-;@ :B;@.9@?:-;@ C8GG@.< GE. <;E.:BF <08GG>@ E8;68; ?-;-A I0@<@ C8GG@.< -.@ 8<@<;E.@ :B;@.9@?:-;@ .@<8>;< EG -FF.@F-;:EB< :B -??:;:EB ;E C8GG@.:BF ?-;- ;0-; :< F;E C@ ?:.@/;>M E8;68; -< 6-.; EG ;0@ <08GG>@A

P<@. /E?@5 &6-.7 @J@/8;@< -.C:;.-.M 8<@. /E?@K <E 8<@. G8B/;:EB< /-B ;0@9<@>L@.@n8:.@ <8C<;-B;:-> [email protected] NE. :B<;-B/@K :G - 8<@. -66>:/-;:EB ->>E/-;@< >-.F@ -E. E;0@. ECb@/;<K ;0@<@ D:>> /EB;@B; GE. [email protected]>> [email protected] 8<-F@A P<@. /E?@ 0-< -/to everything “left” in the JVM heap after the space for Q!! <;E.-F@ -B? <08GG><;E.-F@ -.@ ->>E/-;@?A

3A ).@-;@ -B Q!!



1A %8; :; :B;E /-/0@

OA +EE7 -; 0'$-B@&5+,=+ >EF<EB ;0@ ?.:L@. 6.EF.-9 E.&6-.7 PZ

]D(Z `*&9B.$5$?,-.$:+,-Q9+&-[ Q%%,% -%%XcXN )5 ;,;&-6 &5 ;7BK*&9$*[aceNN I:)i,[ dNdKa W`R 3-

E2)8 V'EE "$EE D25 .2V &5C. &$&2#D $%C.-%#"'"'2( '8 C2(85&'()T V.'C. D25 C%(%))#$)%"$ "2 )$" ".$ "2"%E 8'_$ 2X ".$ @99



&@.:->:j-;:EB :< 8<@? D0@B5

I.-B<G@..:BF ?-;- EL@. ;0@ B@;DE.7



&6:>>:BF ?-;- ;E ?:<7

)-/0:BF ;E [email protected] <@.:->:j@?

H.E-?/-<;:BF L-.:-C>@<

\-L- <@.:->:j-;:EB k.ME <@.:->:j-;:EBL<A



• P<@< Java’s Z7V,9+Z2+'2+0+-,$; [email protected]

• lE.7< D:;0 -BM />-<< ME8 /.@-;@ ;0-; :96>@9@B;<V$E$K)&K0,-)$*)i$7*,

• [E8 /-B /EB;.E> ;0@ [email protected]/@ EG <@.:->:j-;:EB9E.@ />E<@>M CM @J;@B?:BF V$E$K)&K>=+,-5$*)i$7*,

• N>@J:C>@K C8; n8:;@ <>ED

• +@-?< ;E >-.F@ <@.:->:j@? GE.9-;< GE. 9-BM />-<<@<

• Q@/E99@B?@? <@.:->:j-;:EB GE. 6.E?8/

• P<@ kM.E L@.<:EB 1 GE. <6@@?M <@.:->:j-;:E9E.@ /E96-/;B@<<

• !E@< BE; <866E.; ->> 0,-)$*)i$7*, ;M6@

• Q@n8:.@< ME8 ;E 6'2#*.'6 the classes you-?L-B/@

• ZG <@;K D:>> C@ 8<@? GE. <@.:->:j:BF <08GG>@ ?BE?@< -B? -><E <@.:->:j:BF Q!!< ;E ?:<7

C2(XI8$"(“8-%#cI8$#'%E'_$# ”, d2#)I%-%C.$I8-%#cI8$#'%

IE .@F:<;@. ME8. EDB /8<;E9 />-<<@< D:;0 k.ME 8<@ ;0@



IE .@F:<;@. ME8. EDB /8<;E9 />-<<@< D:;0 k.MEK 8<@ ;0@-,?):+,-W-6&@*$::,: 9@;0E?5

E$* 9&53 F 5,H 0'$-B@&53IJK:,+.$:+,-IKKKJK:,+Q''D$;,IKKKJ

9&53K-,?):+,-W-6&@*$::,:I0,jI9*$::Z3M.6@*$::NPR 9*$::Z3M.6@*$::OP

E$* :9 F 5,H 0'$-B@&5+,=+I9&53J

] ZG ME8. ECb@/;< -.@ >-.F@K ME8 9-M B@@? ;E :B/.@-<@ :'$-BKB-6&:,-)$*)i,-K7233,-K;7 /EBG:F 6.E6@.;M

] I0@ ?@G-8>; :< 1K C8; ;0:< L->8@ B@@?< ;E C@ >-.F@ @BE8F0 ;E0E>? ;0@ $B62'*. ECb@/; ME8 D:>> <@.:->:j@A



A A A !"!""

#

!""

!"!""

#

!""

V:F0 /08.B +ED /08.B

)E<; EG Y) :< 6.E6E.;:EB-> ;E ;0@ s\-L- ECb@/;<



A A A !"!""

#

!""

!""

V:F0 /08.B

\-L- ECb@/;<

W<E 8<@ -B -..-M EG ]5+: :B<;@-? E\)5B,%\):+X

UE,-7&:,[?9 Ukk[l/-)5+A@",+$)*: Ukk[l/-)

IE 9@-<8.@ Y) :96-/;5

%-.->>@> $>? Y) ),& Y) Y%-.->>@> Y)



<//;fg8$>%#%EE$EhE1,= <//;fg8$=2(C?%#c!V$$-,= <//;fg

] P<@< 98>;:6>@ ;0.@-?<;E ?E CE;0 ME8BF F@B-B? E>? F@B Y)

] '><E - 98>;:;0.@-?:BF/E96-/;:BF /E>>@/;E.

] VE;&6E; ?E@<

/E96-/;:EB EB>M :BE>? F@B

<//;fg8$>%#%EE$E,=

] P<@< 98>;:6>@ ;0.@-?< ;E?E ME8BF F@B Y)

] l:>> ?@G-8>; ;E &@.:-> EB<:BF>@ /E.@ 9-/0:B@<

] Aka “throughputcollector”

] YEE? GE. D0@B - >E; EGDE.7 :< B@@?@? -B?>EBF 6-8<@< -.@-//@6;-C>@

] P<@ /-<@<5 C-;/06.E/@<<:BF

<//;>%#%EE$E,=B.#$%18[ijk <//;>%#%EE$E=?!B.#$%18[ijk

] )EB/8..@B; ,-.7&D@@6 -7-“Concurrent lowpause collector”

] I.:@< ;E 9:B:9:j@6-8<@< ?8@ ;E Y) CM?E:BF 9E<; EG ;0@ DE.7

/EB/8..@B;>M D:;0-66>:/-;:EB ;0.@-?<

] P<@< <-9@ ->FE.:;09EB ME8BF F@B -<6-.->>@> /E>>@/;E.

] P<@ /-<@<5

] Y-.<;-.

] !@<.@6

] Z< - -B?

/E9Y)



m

&;-F@ 3

IC2EE$C"Y ZI-<7 s3I-<7 s1I-<7 sO



&;-F@ 1

&;-F@ O

&;-F@ 4

AA

\EC s3

AA

&;-F@ S

Q!! $Cb@/;< !'Y &/0@?8>@. I-<7 &/0@?8>@.



I-<7&/0@?8>@.

H

!%%NKV&)5I-%%OJ

K ?-&2'`6I…J

K3)*+,-I…J

] H8:>? [email protected];E. !'Y

] &6>:; F.-60 :B;E

*.B2'* EG ;-<7<

] &8C9:; @-/0 <;-F@ -<.@-?M

]

]

!'Y I-<7&@; I-<7

E3'01,2- ,007&(),0(1

Doesn’t know

)90D, 1,)3&1&;-F@G-:>@?

] +-8B/0@<

:B?:L:?8-> ;-<7<

] Q@;.M G-:>@? E.<;.-FF>:BF ;-<7<



“One EG ;0@ /0->>@BF@< :B 6.EL:?:BF Q!!< -< -B -C<;.-/;:EB :</0EE<:BF - .@6.@<@B;-;:EB GE. ;0@9 ;0-; /-B ;.-/7 >:B@-F@ -/.E<< -D:?@ .-BF@ EG ;.-B<GE.9-;:EB<.”



“The 9E<; :B;@.@<;:BF n8@<;:EB :B ?@<:FB:BF ;0:< :B;@.G-/@ :< 0ED ;E.@6.@<@B; ?@6@B?@B/:@< C@;D@@B Q!!<.”

“We GE8B? :; CE;0 <8GG:/:@B; -B? 8<@G8> ;E />-<<:GM ?@6@B?@B/:@<:B;E ;DE ;M6@<5• B-..ED ?@6@B?@B/:@<K D0@.@ @-/0 6-.;:;:EB EG ;0@ 6-.@B; Q!!

:< 8<@? CM -; 9E<; EB@ 6-.;:;:EB EG ;0@ /0:>? Q!!• D:?@ ?@6@B?@B/:@<K D0@.@ 98>;:6>@ /0:>? 6-.;:;:EB< 9-M

?@6@B? EB :;.”



"J-96>@< EG B-..ED -B? D:?@ ?@6@B?@B/:@<A

"-/0 CEJ :< -B Q!!K D:;0 6-.;:;:EB< <0EDB -< <0-?@? .@/;-BF>@<A

;< 5<



! #$#%&' ($)*+*+,-

! .//

0,+-

1+2*&)

3),4(56

7*$3& 8

7*$3& 9

7*$3& :

=< /< ><

?<

@$(

! 2,A* ($)*+*+,-

“This ?:<;:B/;:EB :< 8<@G8> GE. ;DE .@-<EB<5

!@6@B?@B/:@<5 (-..ED L< l:?@



3X (-..ED ?@6@B?@B/:@< ->>ED GE. 6:6@>:B@? @J@/8;:EB EB EB@ />8<;@. BE?@K

D0:/0 /-B /E968;@ ->> ;0@ 6-.@B; 6-.;:;:EB<A NE. @J-96>@K EB@ /-B -66>M -9-6 GE>>ED@? CM - G:>;@. EB -B @>@9@B;]CM]@>@9@B; C-<:<A

ZB /EB;.-<;K D:?@ ?@6@B?@B/:@< .@n8:.@ ?-;- G.E9 ->> 6-.@B; 6-.;:;:EB< ;E C@-L-:>-C>@ -B? ;E C@ <08GG>@? -/.E<< ;0@ BE?@< 8<:BF - ,-6Q@?8/@]>:7@[email protected];:EBA

1X Q@/[email protected] -G;@. - BE?@ G-:>8.@ :< 9E.@ @GG:/:@B; D:;0 - B-..ED ?@6@B?@B/MK -

EB>M ;0@ >E<; 6-.@B; 6-.;:;:EB< B@@? ;E C@ .@/E968;@?K -B? ;0@M /-B C@.@/E968;@? :B 6-.->>@> EB ?:GG@.@B; BE?@<A ZB /EB;.-<;K :B - >:B@-F@ F.-60 D:;0D:?@ ?@6@B?@B/:@<K - <:BF>@ G-:>@? BE?@ 9:F0; /-8<@ ;0@ >E<< EG <E9@ 6-.;:;:EBG.E9 ->> ;0@ -B/@<;E.< EG -B Q!!K .@n8:.:BF - /E96>@;@ .@]@J@/8;:EB.”

IE ?:<6>-M ;0@ >:B@-F@ EG -B Q!!K &6-.7 6.EL:?@< - $5\%M:3-$&#63 9@;0E?5



)+,(,W #68:$*$5\%M:3-$&#63

&%)t?N -$&#63 B/S0 >,$,*$%4$ L,88%>J\\qSnSs ,$ $%4$I#(% ,$ ^+5w >,$,*$%4$ D,>558J\\qSnes ,$ $%4$I#(% ,$ ^+56

)+,(,W +5:6$)*$5\%M:3-$&#63&%)thN -$&#63 B/S0 -O:11(%>J\\qSnos ,$ &%>:+%;<a%< ,$ ^+56)5(%

xc/S0 L,88%>J\\qSn?s ,$ .,8 ,$ ^+56)5(%WNepw I#($%&%>J\\qSnhs ,$ 1#($%& ,$ ^+56)5(%WNew L,88%>J\\qSngs ,$ .,8 ,$ ^+56)5(%WNe?w >,$,*$%4$ L,88%>J\\qSnSs ,$ $%4$I#(% ,$ ^w >,$,*$%4$ D,>558J\\qSnes ,$ $%4$I#(% ,$ ^

VED ?E ME8 7BED :G - <08GG>@ D:>> C@ /->>@? EB - I.-

.@6-.;:;:EB bE:B /EF.E86 -B? -BM EG ;0@ mHM E. mHMk@M ;.-B<GE



D0.' .+B. 6'(B6.#.#0- E)*. /B$$* /0B$'*' 3F G6)'H

] .@6-.;:;:EB K bE:BK /EF.E86K -B? -BM EG ;0@ mHM E. mHMk@M ;.-B<GE/-B .@<8>; :B <08GG>@<

] ZG ME8 ?@/>-.@ - B89%-.;:;:EB< parameter, it’ll probably shuffle

] ZG - ;.-B<GE.9-;:EB /EB<;.8/;< - <08GG>@?Q!!, it’ll probably shuf

] /E9C:B@HMk@M /->>< - <08GG>@ W<E ?E E;0@. ;.-B<GE.9-;:EB< >:[email protected]@MK D0:/0 -/;8->>M @B? 86 /->>:BF /E9C:B@HMk@MX

%,3 -,'$-+)+)&5I52;/$-+)+)&5:[ ]5+JI);'*)9)+

&-%[ Z-%,-)5?MCP F 52**J[ !""MCP F n

9&$*,:9,I52;/$-+)+)&5:R :1233*, F +-2,J

o

Q!!A</->-

VED ?E ME8 7BED :G - <08GG>@ D:>> C@ /->>@? EB - I.-



Transformations that use “B89%-.;:;:EB<” like distinct will probably shuffle:

%,3 %):+)59+I52;/$-+)+)&5:[ ]5+JI);'*)9)+ &-%[ Z-%,-)5?MCP F

52**J[ !""MCP F

;$'I= FS I=R 52**JJK-,%29,`6W,6II=R 6J FS =R

52;/$-+)+)&5:JK;$'IXKXNJ

] 'B @J;.- 6-.-9@;@. ME8 /-B 6-<< - 7=L ;.-B<GE.9-;:EB ;E >@; &6-.7 ;0-; ME8 D:>> BE; C@ 9@<<:BF D:;0 ;0@ 7@M< -; ->>

] '>> [email protected];:EB< ;0-; <08GG>@ ?-;- EL@. B@;DE.7 D:>> C@B@G:; G.E9 6-.;:;:



6 6

] [email protected];:EB< ;0-; C@B@G:; G.E9 6-.;:;:EB:BF59&?-&2'R ?-&2'Y)+1R V&)5R *,3+Z2+,-8&)5R -)?1+Z2+,-8&)5R ?-&2'`6W,6R

-,%29,`6W,6R 9&;7)5,`6W,6R *&&B2'R K K K

0;;6<5==F:;08CA/E9=-6-/0@=<6-.7=C>EC=9-<;@.=/E.@=<./=9-:B=</->-=E.F=-6-/0@=<6-.7=.??=Q!!A</->-s+O21

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/rdd/RDD.scala#L302





+:

http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/

http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/



VED 9-BM &;-F@< D:>> ;0:< /E?@ .@n8:.@e



:9K+,=+()*,IL:&;,()*,K+=+LJK

;$'I;$'(259JK

3*$+.$'I3*$+.$'(259JK

3)*+,-I3)*+,-(259JK

9&25+IJ

VED 9-BM &;-F@< D:>> ;0:< !'Y .@n8:.@e



VED 9-BM &;-F@< D:>> ;0:< !'Y .@n8:.@e





dd d

!"

:

J o 4

:



!"

!"

J o 4

:

J o 4 J o 4

J o 4

:

• H.E-?/-<; L-.:-C>@< – &@B? - >-.F@ .@-?]EB>M >EE786 ;-C>@ ;E ->> ;0<@B? - >-.F@ G@-;8.@ L@/;E. :B - ,+ ->FE.:;09 ;E ->> BE?@<



• '//898>-;E.< – /E8B; @L@B;< ;0-; E//8. ?8.:BF bEC @J@/8;:EB GE. ?68.6E<@<A "J-96>@5 VED 9-BM >:B@< EG ;0@ :B68; G:>@ D@.@ C>-B7e $/E..86; .@/E.?< D@.@ :B ;0@ :B68; ?-;-<@;e

dd d

<7)(O 1D770(,1 F ,?7&1 0A 1.)(&> U)(2)9*&1J



•

H.E-?/-<; L-.:-C>@< – ->>ED< ME8. 6.EF.-9 ;E @GG:/:@B;>M <@B? - >-.FL->8@ ;E ->> ;0@ DE.7@. BE?@< GE. 8<@ :B EB@ E. 9E.@ &6-.7 [email protected];:E<@B?:BF - >-.F@K .@-?]EB>M >EE786 ;-C>@ ;E ->> ;0@ BE?@<A

• '//898>-;E.< – ->>ED< ME8 ;E -FF.@F-;@ L->8@< G.E9 DE.7@. BE?@;0@ ?.:L@. 6.EF.-9A )-B C@ 8<@? ;E /E8B; ;0@ s EG @..E.< <@@B :B ->:B@< <6.@-? -/.E<< 322< EG BE?@<A $B>M ;0@ ?.:L@. /-B -//@<< ;0@ L

-//898>-;E.K ;-<7< /-BBE;A NE. ;-<7<K -//898>-;E.< -.@ D.:;@]EB>MA

dd d

H.E-?/-<; L-.:-C>@< >@; 6.EF.-99@. 7@@6 - .@-?]EB>M L-.:-C>@ /-/0@? EB @-/0 9-/0:B@ .-;0@. ;0-B<0:66:BF - /E6M EG :; D:;0 ;-<7<



<0:66:BF - /E6M EG :; D:;0 ;-<7<

NE. @J-96>@K ;E F:L@ @[email protected] BE?@ - /E6M EG - >-.F@:B68; ?-;-<@; @GG:/:@B;>M

&6-.7 -><E -;;@96;< ;E ?:<;.:C8;@ C.E-?/-<; L-.:-C>@<8<:BF @GG:/:@B; C.E-?/-<; ->FE.:;09< ;E .@?8/@/E998B:/-;:EB /E<;

E,( M&5,>+,)$U,& B )+*M&5,>+,)$/_&&,</eQ SQ g00

&/->-5



M&5,>+,)$U,&*E,(:%

M&5,>+,)$U,& B )+*M&5,>+,)$/(#)$/&,63%/eQ h000

M&5,>+,)$U,&*E,(:%

%M;0EB5



+:B7

http://www.mosharaf.com/wp-content/uploads/orchestra-sigcomm11.pdf

http://www.mosharaf.com/wp-content/uploads/orchestra-sigcomm11.pdf



!"

!"#$%&'(

P<@< VII%



!"

!"

12 ,H G:>@

!"

P<@< C:;;E..@B;



!"

!"

12 ,H G:>@

A A A! #$ ! #$ ! #$ ! #$



!"!" !"



!"!"

!" !"!"

A l t i bl th t l b “ dd d” t th h

dd d



Accumulators are variables that can only be “added” to through

-B B**0/#B.#I' [email protected];:EB

P<@? ;E :96>@9@B; /E8B;@.< -B? <89<K @GG:/:@B;>M :B 6-.->>@>

&6-.7 B-;:L@>M <866E.;< -//898>-;E.< EG B89@.:/ L->8@ ;M6@< -B?<;-B?-.? 98;-C>@ /E>>@/;:EB<K -B? 6.EF.-99@.< /-B @J;@B?GE. B@D ;M6@<

Only the driver program can read an accumulator’s value, not the;-<7<

!"# !""#$ $ %"%"&&'('#")*+,&-

%"%'!(!))*)+,*,.++"/,-0 .0 /0 0--%12(*!"3,4 $1 !""#$ 2$

&/->-5dd d



!""#$%!"#'3

,++:. : )+6#11242+#"3$=f>&>> : )+68,&,((%(#[%/qeQ SQ gQ hs05'? ?/40N

0+3,#+ ,++:.,++:. @: 4

&>>615&%,+O/10

,++:.6A#+2'

%M;0EB5



%M&6-.7 -; - Y>-B/@



l.:;@ &6-.7 bEC<:B %M;0EB

Q8B :B;@.-/;:L@ bEC< :B ;0@ <0@>>

&866@J;@

%M&6-.7

52 @,A.=

802BB ACD

604BB DCEE.F>=



&6-.7 )E.@ "BF:B@W&/->-X

&;-B?->EB@ &/0@?8>@. ['Q( ,@<E+E/->

\-L- '%Z

%M&6-.7

9:;+<

GCF>.">

GCF>+CAA.+

9:;+<

GCF>.">

1H5I

9CD<.>

1,:.]44

,+>:CK &_+K <08GG>@



+E/-> !:<7

!.:L@. \#,

"J@/8;E. \#,

"J@/8;E. \#,

1,:.

Z0(O&( C)-.4(2U&( C)-.2'&

(I=J

]44

,+>:CK &_+K <08GG>@

!-;- :< <;E.@? -< %:/7>@? ECb@/;< :B -B Q!!q'..-MqHM;@`)>007]44



C)77&>]44

/?,.0']44 Q!!q'..-Mq r rK K K

W322 kH – 3,H @-/0 6:/7@? ECb@/;X

)%M;0EBW?@G-8>; 6M;0EBX

)0EE<@ [E8. %M;0EB Z96>@9@B;-;:EB

9:;+<

GCF>.">



6M6M• \ZIK <E G-<;@. • >@<< [email protected]• )NNZ <866E.;

GCF>.">

4(2U&( C)-.2'&Z0(O&( C)

h /_0/Q!WX"!]<>!X/_C#ZDF'6'6 /_0/Q!WX/_C#ZDF'6'6 KT7

h /_0/Q!WX"!]<>!X/_C#ZDF'6'6 /_0/Q!WX/_C#ZDF'6'6 KT7)5T:'$-BU:27;)+

$Q

T09 I/?,.0' FN! /?/? FNaNK <7&&> D7

lE.? )E8B; S3 < 34 < 1AT J

&E.; Sc < SS < 3A24 J

&; ; 3TS O c Si

I0@ [email protected]/@ <6@@? 86 D:>> ?@6@B? EB DE.7 >E-? WG.E9 12t ;E O2

V@.@ -.@ <E9@ C@B/09-.7<5



&;-;< 3TS < OAc < Si J

V@.@ :< ;0@ /E?@ 8<@? GE. C@B/09-.75

-%% F :9K+,=+()*,IL+,=+LJ

%,3 H&-%9&25+IJ[

-%%K3*$+.$'I*$;7%$ =[=K:'*)+IpTpJJq

K;$'I*$;7%$ =[I=RNJJK-,%29,`6W,6I*$;7%$ =R6[=l6JK9&**,9+Q:.

%,3 :&-+IJ[-%%K:&-+`6I*$;7%$ =[=R NJK9&25+IJ

%,3 :+$+:IJ[

:9K'$-$**,*)i,I-$5?,INcOrJR OcJK3*$+.$'I*$;7%$ =[ =-$5?,IacOrJJ

0;;6<5==F:;08CA/E9=-6-/0@=<6-.7=68>>=13SS





&6-.7 <E.;@? ;0@ <-9@ ?-;- ab A)1,&(8<:BF KGb A&c&

;0-B

322IH !-M;EB- &E.; )E96@;:;:EB 123S

'>> ;0@ <E.;:BF ;EE7 6>-/@ EB ?:<7 WV!



lE.7 CM !-;-C.:/7< @BF:B@@.<5 Q@MBE>? u:BK %-.L:j !@M0:9K u:-BF.8: ,@BFK '>: Y0E?<:K ,-;@

,E.@ :BGE5

0;;65==<E.;C@B/09-.7AE.F

0;;65==?-;-C.:/7<A/E9=C>EF=123S=33EGG:/:->>M]<@;<]-]B@D].@/E.?]:B]>-.F@]<

using Spark’s in][email protected] /-/0@h



] &;.@<<@< “shuffle” which underpins everything from SQL to ,>>:C

] &E.;:BF :< /0->>@BF:BF C=/ ;0@.@ :< BE .@?8/;:EB :B ?-;-



] &E.; 322 IH o 422 IH ?:<7 Z=$ -B? 122 IH B@;DE.7

B'32'&&(2'3 6'U&1,5&', 2' <7)(OJ

] &E.;]C-<@? <08GG>@ W&%'Qk]12S4X] (@;;M B-;:L@ B@;DE.7 ;.-B<6E.; W&%'Qk]1SciX] "J;@.B-> <08GG>@ <@.L:/@ W&%'Qk]OTgcX

I*&U&( E77*2-),20' *&U&* :&-.'2dD&1J

] Y) -B? /-/0@ G.:@B?>M [email protected] >-ME8;] %:6@>:B:BF

!"!""

#

!""

I

I

] ZB;@> u@EB )%P "4 1cT2 a 1A4 YVj D= O1 /E.@<] 1SS YH EG Q',] i J i22 YH &&! -B? Q'Z! 2 <@;86 GE.9-;;@? D:;0 =@J;S] UgA4 YC6< W?J? 9K(*X C-B?D:?;0 C@;D@@B 1 .-B?E9 BE?@<

] "-/0 .@/E.?5 322 CM;@< W32 CM;@ 7@M ` g2 CM



")15 :1AiJ>-.F@

W12c DE.7@.<X

] $6@B\!k 3AT

] V!N& 1ASA3 D= <0E.; /:./8:; >E/-> .@-?< @B-C>@

] '6-/0@ &6-.7 3A1A2

] &6@/8>-;:L@ "J@/8;:EB EGG

] ZB/.@-<@? +E/->:;M l-:; ;E :BG:B:;@

] )E96.@<<:EB ;8.B@? EGG GE. :B68;K E8;68; ` B

] P<@? PB<-G@ ;E 68; ->> ;0@ ?-;- EGG]0@-6 -B:; 9-B8->>M W:A@A B@L@. ;.:FF@.@? ;0@ Y)X

] O1 <>E;< 6@. 9-/0:B@] cK4g1 <>E;< ;E;->

?-&2'`6W,6



o :&-+`6W,6

-,%29,`6W,6

:'$-BK:1233*,K:')**F3$*:,



W'GG@/;< .@?8/@. <:?@ -B? 7@@6< ->> ;0@ ?-;- :B [email protected]

] ,8<; ;8.B ;0:< EB GE. ?MB-9:/



] lE.7@. \#, <@.L@< G:>@<] (E?@ ,-B-F@. <@.L@< G:>@<

] l-< <>ED C@/-8<@ :; 0-? ;E /E6M ;0@ ?-;- O ;:9@<



,-6 E8;68; G:>@EB >E/-> ?:.

+:[email protected]@>C8GG@.

!"

(Z)C8GG@

] P<@< - ;@/0B:n8@ /->>@? [email protected]]/E6M

] Z< - 9-6]<:?@ E6;:9:j-;:EB ;E <@.L@ ?-;- [email protected]:/7>M ;E .@n8@<;:BF .@?8/@.<



,-6 E8;68; G:>@EB >E/-> ?:.

(Z)C8GG@.

&;:JK &;:JK &;:JK &;:JK

] "B;:CM Z=$V!N&

w 32K222 .

] (E;:/@ ;0-; 9-60-< ;E 7@@6 O G:>@0-B?>@< E6@B

[ L :E2Cc8



/.LMD.JK /.LMD.JK /.LMD.JK

>E/-

] ,E<

0-B?>@< E6@B

I:9&E.;

&;:JK &;:JK &;:JK &;:JK

W1iK222 8B:n8@ C>E/7<X

QN o 1

1

o OAc YH



] $B>M EB@ G:>@ 0-B?>@E6@B -; - ;:9@

&;:JK &;:JK &;:JK &;:JK

] 4 D-] 4 D-

1

W1iK222 8B:n8@ C>E/7<X

QN o 1



/.LMD.JK /.LMD.JK /.LMD.JK

QN o 1

C&(3&<0

I:9&E.;

]

]





@1&(64 ^)5& E3& 80-),20' /&,

1iSg1gS1 \E0B Y->; O1 (@D [E.7 &@- VE.<@

g4i1gO1S l:B<;EB &9:;0 S3 $/@-B:- 'B;

g1iT3Tc3 IE9 &-DM@. 3T ,:<<:<<:66: Q-//EEB



OT4iSgO1 )-.>E< V:BEbE<- OO $.>-B?E )-;

TOcSi1TS +8:< QE?.:F8@j OS $.>-B?E !EF<

\!H)=$!H) [E8. '66



091,;$!""] Q!! EG QED ECb@/;<K @-/0 .@6.@<@B;:BF - .@/E.?

] QED ECb@/;< o ;M6@ d /E>A B-9@ EG @-/0

] &;E.@< ?-;- [email protected] @GG:/:@B;>M CM ;-7:BF -?L-B;-F@ EG ;0@ </0@

http://hive.apache.org/

http://hive.apache.org/



] &/0@9-Q!!< -.@ -><E .@F8>-. Q!!<K <E ME8 /-B .8B;.-B<GE.9-;:EB< >:7@ 9-6WX E. G:>;@.WX

] '>>ED< B@D [email protected];:EB<K >:7@ .8BB:BF &_+ EB ECb@/;<

0;;6<5==?-;-C.:/7<A/E9=C>E

F=1234=21=3T=:B;.E?8/:BF]?-;-G.-9@<]:B]<6-.7]GE.]>-.F@]</->@]?-;-]</:@B

https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html





l : h



l-.B:BFh

$B>M >EE7< -; G:.<; .ED









!"#$$%&'$#()*+&%,$%-$&%,./***0*1#($%&/2*3%$!%4$*+56$,#6)/7-8,&9700*+5:6$;<=#6>5"/-%+56>)/?00



k-G7-

N>89@

I)% <E/7@;] &/->-C>@] V:F0];0.E8F068;] N-8>;];E>@.-B;



V!N&

&O

k:B@<:<

ID:;;@. )E96>@J ->FE.:;09< /-B C@ @J6.@<<@? 8<:BF5] &6-.7 ;.-B<GE.9-;:EB<5 ;$'IJR -,%29,IJR V&)5IJR ,+9] ,+>:C d Y.-60u] &_+

H-;/0 Q@->;:9@



$B@ 8B:G:@? '%Z

I-;0-F-;- !-< WI!X

] +@-? ?@L@>E6@. EG &6-.7 &;.@-9:BF d )E99:;;@.EB '6-/0@ &6-.7 /E.@

] V@>6@? .@]D.:;@ &6-.7 )E.@ :B;@.B->< :B 1231 ;E9-7@ :; 32J G-<;@. ;E <866E.; &;.@-9:BF 8<@ /-<@<

] $B >@-L@ G.E9 P) [email protected]@>@M %0! 6.EF.-9

] "J5 ZB;@.B a '9-jEB ZB;@.B a )EBL:L- Q@<@-./0



"J5 ZB;@.B a '9-jEBK ZB;@.B a )EBL:L-K Q@<@-./0'<<:<;-B; a ,:/.E<EG; Q@<@-./0 ZB?:-

] 3 F8Mx ?E@< BE; </->@

] &/->@< ;E 322< EG BE?@<

] H-;/0 <:j@< -< <9->> -; 0->G - <@/EB?

] %.E/@<<:BF >-;@B/M -< >ED -< 3 <@/EB?

] "J-/;>M]EB/@ <@9-B;:/< BE 9-;;@. D0-; G-:><

W>:L@ <;-;:<;:/<X



%-F@ L:@D< k-G7- GE. C8GG@.:BF &6-.7 G

&9-.; 9@;@. .@-?:BF<W'BE9->M !@;@/;:EBX



+:L@ D@-;0@. ?-;-

\E:B 1 >:L@ ?-;-<E8./@<

ZB68; ?-;- <;.@-9

H-;/0@< @[email protected] u <@/EB?<



W!:</.@;:j@? &;.@-9XH-;/0 :B;@.L-> o 4 <@/EB?<

I o 2 I o d

ZB68;

!&;.@-9



H>E/7 s3

Q!! a Io2

H>E/7 s1 H>E/7 sO H>E/7 s3

Q!! a Iod4

H>E/7 s1 H>E!&;.@-9

$B@ Q!! :< /.@-;@? @[email protected] 4 <@/EB?<

H>E/7 s3 H>E/7 s1 H>E/7 sO

4 <@/ ,-;@.:->:j@h



%-.;A s3 %-.;A s1 %-.;A sO

%-.;A s3 %-.;A s1 %-.;A sO

H&-%:!""

3*$+.$'IJ

*)5,:!""

3-&; '6:'$-B );'&-+ 0'$-B@&5+,=+

3-&; '6:'$-BK:+-,$;)5? );'&-+ 0+-,$;)5?@&5+,=+

2 40/#$/ # +5.#+ %$0/#6-78457$/&$ 9-$1 $95 950:-78 $10/#; #7; <#$.1 -7$/0=#+ 5> ? ,/.57;

:9 F 0'$-B@&5+,=+IL*&9$*MOPLR LD,+H&-BY&-%@&25+LJ

::9 F 0+-,$;)5?@&5+,=+I:9R aJ

2 40/#$/ # "%$0/#6 $1#$ 9-++ .577/.$ $5 15,$7#6/@A50$B +-:/ +5.#+15,$@CCCC

*)5,:"0+-,$; F ::9K:&9B,+C,=+0+-,$;IL*&9$*1&:+LR ssssJ

% +-$ 1 +- - $ ;



2 %A+-$ /#.1 +-7/ -7$5 950;,

H&-%:"0+-,$; F *)5,:"0+-,$;K3*$+.$'I*$;7%$ *)5,[ *)5,K:'*)+IL LJJ

2 45D7$ /#.1 950; -7 /#.1 <#$.1

'$)-:"0+-,$; F H&-%:"0+-,$;K;$'I*$;7%$ H&-%[ IH&-%R NJJ

H&-%@&25+:"0+-,$; F '$)-:"0+-,$;K-,%29,`6W,6I*$;7%$ =R 6[ = l 6J

2 E0-7$ $1/ >-0,$ $/7 /+/6/7$, 5> /#.1 )"" 8/7/0#$/; -7 $1-, "%$0/#6 $5 $1/ .57,5+/

H&-%@&25+:"0+-,$;K''-)5+IJ

::9K:+$-+IJ 2 %$#0$ $1/ .56AD$#$-57

::9K$H$)+C,-;)5$+)&5IJ 2 F#-$ >50 $1/ .56AD$#$-57 $5 $/06-7#$/

[email protected]:B-> s3 [email protected]:B-> s1



h 59 U*B ssss

1,**& H&-*%

h KT5,+H&-BXH&-%9&25+K'6 *&9$

K K K

UUUUUUUUUUUUUUUUUUUUUUUUUU

C);,[ OcNaUcrUOa Na[Oa[ON

UUUUUUUUUUUUUUUUUUUUUUUUUU

I1,**&R OJIH&-*%R NJ

!"

/**0 12

#

/**0 13

NACD<0 12

:

: :

: : !"

/**0 14

#

: :

H-;/0 :B;@.L-> o c22 9<

]



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 15

NACD<0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

!"

/**0 12

#

/**0 13

NACD<0 12

: ]

: :

: :

!"

/**0 14

#

/** 15

: :

!

#

NACD<0 13

NACD<0 13

H-;/0 :B;@.L-> o c22 9<



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 15

NACD<0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

!"

/**0 12

#

/**0 13

NACD<0 12

: ]

: :

: : !"

/**0 12

#

/** 13

: :

!

#

NACD<0 13

NACD<0 13

H-;/0 :B;@.L-> o c22 9<

NACD<0 14



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 13

NACD<0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

NACD<0 14

!"

/**0 12

#

/**0 13

/**0 12

: ]

: :

: : !"

/**0 12

#

/** 13

:

: :

:

!

#

/**0 13

/**0 13

H-;/0 :B;@.L-> o c22 9<

/**0 14



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 13

/**0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

/**0 14

!"

/**0 12

#

/**0 13

/**0 12

: ]

: :

: : !"

/**0 12

#

/** 13

:

: :

:

!

#

/**0 13

/**0 13

H-;/0 :B;@.L-> o c22 9<

/**0 14



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 13

/**0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

/**0 14



!"

#

NACD<0 12

: ]

6 , *

: :

: : !"

#

:

: :

:

H-;/0 :B;@.L-> o c22 9<

!

#

NACD<0 12

1 :B



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

NACD<0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

NACD<0 12

!"

#

NACD<0 12

: ]

6',&(')*

: :

: : !"

#

:

: :

:

!

#

NACD<0 12

6 ,

H-;/0 :B;@.L-> o c22 9<

NACD<0 13

NACD<0 14

NACD<0 13

NACD<0 14

NACD<0 13

NACD<0 14

NACD<0 13 NACD<0 14



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

NACD<0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

NACD<0 12

!"

#

/**0 12

: ]

6',&(')*

: :

: : !"

#

:

: :

:

!

#

/**0 12

6',&

H-;/0 :B;@.L-> o c22 9<

/**0 13

/**0 14

/**0 13

/**0 14

/**0 13

/**0 14

/**0 13

/**0 14

,-;@.:->:j@h



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 12

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

: :

6',&:.(&

V< 421O

/**0 12

!"

#

/**0 14

: ]

6',&(')*

: :

: : !"

#

:

: :

:

!

#

/**0 14

6',&

H-;/0 :B;@.L-> o c22 9<

/**0 15

/**0 1?

/**0 13

/**0 13

/**0 1?

/**0 12

/**0 12

/**0 16

PB:EBh



*+,-.+

6',&(')*:.(&)>1

&&! &&!V< 421O

/**0 15

6',&(')*:.(&)>1

&&! &&!V< 421O

: :

6',&:.(&

V< 421O

/**0 16

] N:>@ <M<;@9<] &E/7@; )EBB@/;:EB<

] '77- '/;E.<

] k-G7-] N>89@

] ID:;;@.

] 'BMD0



&E8./@< ?:.@/;>M -L-:>-C>@:B 0+-,$;)5?@&5+,=+ '%Z

Q@n8:.@< >:B7:BF -F-:B<;@J;.- ?@6@B?@B/:@<

Q@n8:.@< :968<@.]?@G:B@?





;$'I J

3*$+.$'I J

3)*+,-I J

25)&5I &+1,

-,%29,I J

V&)5I 0 M PJ

2'%$+,0+$+,`6W,6I Jm



3)*+,-I J

-,'$-+)+)&5I52;/$-+)+)&5:J

9&25+I

9&25+`6<$*2,IJ

-,%29,Q`6W,6I RM52;C$:B:PJ

V&)5I &+1,-0+-,$;RM52;C$:B:PJ9&?-&2'

+-$5:3&-;I J

/**

/**

2'%$+,0+$+,`6W,6I J

IE 8<@5

5 ->>ED< ME8 ;E 9-:B;-:B -.C:;.-.M <;-;@ D0:>@/EB;:B8E8<>M 86?-;:BF :; D:;0 B@D :BGE.9-;:EBA

IE 9-:B;-:B - .8BB:BF /E8B; EG @-/0 D:B - ;@J; ?-;- <;.@-9 !+'6' 6)--#-2 /0

'$)-: F

m



3X !@G:B@ ;0@ <;-;@Y%( %#:'"#%#D 1%"% "D-$Z

1X !@G:B@ ;0@ <;-;@ 86?-;@ G8B/;:EBY8-$C'XD V'". % X5(C"'2( .2V "2 5-1%"$ ".$ 8"%"$ 58'() ".$

-#$\'258 8"%"$ %(1 ($V \%E5$8 X#2& ".$ '(-5" 8"#$%&Z

%,3 2'%$+,(259+)&5I5,H<$*2,:R -255)5

)3 -255)5?@&25+ ): D&5,[

-255)5?@&25+ F c

-,+2-5 :2;I5,H<$*2,:R -255)5?@&2

^ 5,H E$*2,: H)+1 +1, '-,E

^ +& ?,+ +1, 5,H 9&25+

:B - ;@J; ?-;- <;.@-9 !+'6' 6)--#-2 /0#-.'2'6 .L(' 05 *.B.'A5

-255)5?@&25+: F '$)-:K2'%$+,0+$+,`6W

m Q@n8:.@< - /0@/76E:B; ?:.@/;E.M ;E C@/EBG:F8.@?

NE. @J-96>@5

5 /-B C@ 8<@? ;E -66>M -BM Q!! [email protected];:EB ;0-;:< BE; @J6E<@? :B ;0@ !&;.@-9 '%ZA

:'$;]53&!"" F :9K')9B*,()*,IKKKJ ^ !"" 9&5+$)5)5? :'$;

^ V&)5 %$+$ :+-,$; H)+1 :'$; )53&-;$+)&5 +& %& %$+$ 9*,9*,$5,%"0+-,$; F H&-%@&25+:K+-$5:3&-;I*$;7%$ -%%[

-%%KV&

+-$5:3&-;I J

/**

/**



6

] N8B/;:EB->:;M ;E bE:B @[email protected] C-;/0 :B -?-;- <;.@-9 D:;0 -BE;0@. ?-;-<@; :< BE;?:.@/;>M @J6E<@? :B ;0@ !&;.@-9 '%ZA

] ZG ME8 D-B; ;E ?E .@->];:9@ ?-;-

/>@-B:BF CM bE:B:BF ;0@ :B68; ?-;-<;.@-9 D:;0 6.@]/E968;@? <6-9:BGE.9-;:EB -B? ;0@B G:>;@.:BF C-<@? EB :;A

V

E.

,+>:C Y



H)5%&HI H)5%&H\,5?+1R :*)%,]5+,-E$*J

9&25+`6Y)5%&HI H)5%&H\,5?+1R :*)%,]5+,-E$*J

9&25+`6<$*2,Q5%Y)5%&HI H)5%&H\,5?+1R



-,%29,`6Y)5%&HI R H)5%&H\,5?+1R :*)%,]5+,-E$*J

-,%29,`6W,6Q5%Y)5%&HI R H)5%&H\,5?+1

R :*)%,]5+,-E$*

RM52;C$:B:P

J

-,%29,`6W,6Q5%Y)5%&HI R R H)5%&H\,5?+1R :*)%,]5+,-E$*RM52;C$:B:PJ

'-)5+IJ

:$E,Q:C,=+()*,I'-,3)=R M:233)=PJ

3&-,$9



:$E,Q:Z7V,9+()*,:I'-,3)=R M:233)=PJ

:$E,Q:#$%&&'()*,:I'-,3)=R M:233)=PJ



Date post:	18-Feb-2018
Category:	Documents
Upload:	adarsh-pannu
View:	212 times
Download:	0 times

Spark Advanced DevOps

Documents