DryadInc: Reusing work in large-scale computations

Lucian Popa*+, Mihai Budiu+, Yuan Yu+, Michael Isard+

+ Microsoft Research Silicon Valley* UC Berkeley

Problem Statement

…

…

Goal: Reuse (part of) prior computations to: - Speed up the current job- Increase cluster throughput- Reduce energy and costs

Outputs

Inputs

Distributed Computation

Append-only data

Propose Two Approaches

1. IDEReuse IDEntical computations from the past(like make or memoization)

2. MERDo only incremental computation on the new data and MERge results with the previous ones(like patch)

Context• Implemented for Dryad

– Dryad Job = Computational DAG• Vertex: arbitrary computation + inputs/outputs• Edge: data flows

Simple Example: Record Count

C

I2

C

AAdd

Outputs

Inputs (partitions)

Count

I1

IDE – IDEntical ComputationRecord Count

C

I2

C

AAdd

Outputs

Inputs (partitions)

Count

I1

First executionDAG

IDE – IDEntical Computation

Second executionDAG

Record Count

C

I2

C

AAdd

Outputs

Inputs (partitions)

Count

I1 I3

C

New Input

IDE – IDEntical Computation

Second executionDAG

Record Count

C

I2

C

AAdd

Outputs

Inputs (partitions)

Count

I1 I3

C

Identical subDAG

IDE – IDEntical Computation

IDE Modified DAG

Replaced with Cached Data

Replace identical computational subDAG with edge data cached from previous execution

AAdd

Outputs

Inputs (partitions)

Count

I3

C

IDE – IDEntical Computation

IDE Modified DAG

Use DAG fingerprints to determine if computations are identical

AAdd

Outputs

Inputs (partitions)

Count

I3

C

Replace identical computational subDAG with edge data cached from previous execution

Semantic Knowledge Can Help

C

I2

C

A

I1

Reuse Output

Semantic Knowledge Can Help

C

I2

C

A

I1

C

I3

A Merge (Add)

Previous Output

Incremental DAG

MER – MERgeable Computation

C

I2

C

A

I1

C

I3

A Merge (Add)

Automatically Inferred

Automatically Built

User-specified

MER – MERgeable Computation

C

I2

C

A

I1

A

C

I2

C

A

I1 I3

C

Empty

Save to CacheIncremental DAG – Remove Old Inputs

Merge Vertex

IDE in practice

6 input DAG

IDE 6 (4+2) input DAG

9 input DAG

MER 9 (5+4) input DAG

MER in practice

EmptyEmpty

Evaluation – Running time

0.260.520.781.041.31.561.822.082.342.62.863.123.383.643.94.164.424.684.945.25.465.725.986.246.56.767.027.287.547.88.068.328.588.849.19.360000000000019.629.8810.1410.410.6610.9211.1811.4411.711.9612.2212.4812.741313.2613.5213.7814.0414.314.5614.8215.0815.3415.615.8616.1216.3816.6416.917.1617.4217.6817.9418.218.4618.7218.9819.2419.519.7620.0220.2820.5420.821.0621.3221.5821.8422.122.3622.6222.8823.1423.423.6623.9224.1824.4424.724.9625.2225.4825.742626.2626.5226.7827.0427.327.5627.8228.0828.3428.628.8629.1229.3829.6429.930.1630.4230.6830.9431.231.4631.7231.9832.2432.532.7633.0233.2833.5433.834.0634.3234.5834.8435.135.3635.6235.8836.1436.436.6636.9237.1837.4437.737.9638.2238.4838.743939.2639.5239.7840.0440.340.5640.8241.0841.3441.641.8642.1242.3842.6442.943.1643.4243.6843.9444.244.4644.7244.9845.2445.545.7646.0246.2846.5446.847.0647.3247.5847.8448.148.3648.6248.8849.1449.449.6649.9250.1850.4450.750.9651.2251.4851.745252.2652.5252.7853.0453.353.5653.8254.0854.3454.654.8655.1255.3855.6455.956.1656.4256.6856.9457.257.4657.7257.9858.2458.558.7659.0259.2859.5459.860.0660.3260.5860.8461.161.3661.6261.8862.1462.462.6662.9263.1863.4463.763.9664.2264.4864.746565.2665.5265.7866.0466.366.5666.8267.0867.3467.667.8668.1268.3868.6468.969.1669.4269.6869.9470.270.4670.7270.9871.2471.571.7672.0272.2872.5472.873.0673.3273.5873.8474.174.3674.6274.8875.1475.475.6675.9276.1876.4476.776.9677.2277.4877.747878.2678.5278.7879.0479.379.5679.8280.0880.3480.680.8681.1281.3881.6481.982.1682.4282.6882.9483.283.4683.7283.9884.2484.584.7685.0285.2885.5485.886.0686.3286.5886.8487.187.3687.6287.8888.1488.488.6688.9289.1889.4489.789.9690.2290.4890.749191.2691.5291.7892.0492.392.5692.8293.0893.3493.693.8694.1294.380

200

400

600

800

1000

1200

1400

1600

no cache identical

incremental

data size (GB)

Job

Run

ning

Tim

e (s

) No Cache

IDE

MER

Word Histogram Application – 8 nodes

Discussion

• MapReduce: just a particular case– IDE reuses the output of Mappers – MER requires combined Reduce function

• Combine IDE with MER: benefits don’t add up– IDE can be used for the incremental DAG at MER

• More semantic knowledge: further opportunities– Generate merge function automatically– Improve incremental DAG

Conclusions & Questions

• Problem: reuse work in distributed computations on append-only data

• Two methods:– INC – reuse IDEntical past computations

• No user effort– MER – MERge past results with new ones

• Small user effort, potentially larger gains

• Implemented for Dryad

Backup Slides

Architecture

Modify DAG before run

Dryad Job Manager

Cache Server

IDE/MER Rerun Logic

Update cache after runRUN

Use fingerprints to identify identical computational DAGs• Guarantee that both computation and data are unchanged

First DAG

Store to Cache: <hash, outputs>

Second DAG

IDE – IDEntical Computation

cached

Use a heuristic to identify the vertices to cache

First DAG Second DAG

IDE – IDEntical Computation

…

…Hash Distribute (Map)

Merge Sort

Count (Reduce)

Sort

Outputs

Inputsn

m

Word Histogram Application

Evaluation – Running time

0.260.520.781.041.31.561.822.082.342.62.863.123.383.643.94.164.424.684.945.25.465.725.986.246.56.767.027.287.547.88.068.328.588.849.19.360000000000019.629.8810.1410.410.6610.9211.1811.4411.711.9612.2212.4812.741313.2613.5213.7814.0414.314.5614.8215.0815.3415.615.8616.1216.3816.6416.917.1617.4217.6817.9418.218.4618.7218.9819.2419.519.7620.0220.2820.5420.821.0621.3221.5821.8422.122.3622.6222.8823.1423.423.6623.9224.1824.4424.724.9625.2225.4825.742626.2626.5226.7827.0427.327.5627.8228.0828.3428.628.8629.1229.3829.6429.930.1630.4230.6830.9431.231.4631.7231.9832.2432.532.7633.0233.2833.5433.834.0634.3234.5834.8435.135.3635.6235.8836.1436.436.6636.9237.1837.4437.737.9638.2238.4838.743939.2639.5239.7840.0440.340.5640.8241.0841.3441.641.8642.1242.3842.6442.943.1643.4243.6843.9444.244.4644.7244.9845.2445.545.7646.0246.2846.5446.847.0647.3247.5847.8448.148.3648.6248.8849.1449.449.6649.9250.1850.4450.750.9651.2251.4851.745252.2652.5252.7853.0453.353.5653.8254.0854.3454.654.8655.1255.3855.6455.956.1656.4256.6856.9457.257.4657.7257.9858.2458.558.7659.0259.2859.5459.860.0660.3260.5860.8461.161.3661.6261.8862.1462.462.6662.9263.1863.4463.763.9664.2264.4864.746565.2665.5265.7866.0466.366.5666.8267.0867.3467.667.8668.1268.3868.6468.969.1669.4269.6869.9470.270.4670.7270.9871.2471.571.7672.0272.2872.5472.873.0673.3273.5873.8474.174.3674.6274.8875.1475.475.6675.9276.1876.4476.776.9677.2277.4877.747878.2678.5278.7879.0479.379.5679.8280.0880.3480.680.8681.1281.3881.6481.982.1682.4282.6882.9483.283.4683.7283.9884.2484.584.7685.0285.2885.5485.886.0686.3286.5886.8487.187.3687.6287.8888.1488.488.6688.9289.1889.4489.789.9690.2290.4890.749191.2691.5291.7892.0492.392.5692.8293.0893.3493.693.8694.1294.380

200

400

600

800

1000

1200

1400

1600no cache

identical

incremental

identical 20ext

incremental 20ext

data size (GB)

Job

Run

ning

Tim

e (s

)

No CacheIDE (4 ext)MER (4 ext)IDE (20 ext)MER (20 ext)

Results similar if 20 more inputs appended ~ 1.2GB