Presentation

Numerical reproducibility for exascale HPC

Chemseddine CHOHRA

Université de Perpignan Via Domitia (UPVD)

16 Juin 2014

Chemseddine CHOHRA (Université de Perpignan Via Domitia (UPVD))Numerical reproducibility for exascale HPC 16 Juin 2014 1 / 67

Introduction and problematic


Limited machine precision.Using �oating point number as approximation.x −→ X = �(x) if x /∈ F or x si x ∈ F.X + Y 6= X ⊕ Y = �(X + Y).

Non-associativity of addition.A ⊕ (B ⊕ C) 6= (A ⊕ B) ⊕ C.For instance : M = 253; (-M ⊕ M) ⊕ 1 6= -M ⊕ (M ⊕ 1)




Figure 1.1 : No reproducibility of summation
















Non-reproducibility of summation on parallel systems.Problem for debuging.Problem for validating results.

Guarantee the reproducibility for BLAS.Level 1 : max, min, scal, axpy, norm, asum, dot.dot can be transformed to a summation problem.




Non-reproducibility of summation on parallel systems.Problem for debuging.Problem for validating results.

Guarantee the reproducibility for BLAS.Level 1 : max, min, scal, axpy, norm, asum, dot.dot can be transformed to a summation problem.



Sommaire

1 Introduction and problematic

2 Solution

3 Optimization

4 Parallelism

5 Conclusion and Work in progress


Solution

Sommaire


2 SolutionAccSumFastAccSumiFastSumHybridSumOnlineExactCompare algorithms

3 Optimization

4 Parallelism



Solution

Solution

Ensure reproducibility.1 Static scheduling and deterministic reduction.2 Demmel and Nguyen's solutions (2013).

Use an exact summation algorithm (Always reproducible).


Solution

Exact summation

How to calculate a RTN (rounded to nearest) sum.But we know how since 1970s.

Several algorithms have been proposed.FastSum (2006).AccSum (2008).FastAccSum (2008).iFastSum (2009).HybridSum (2009).OnlineExact (2010).


Solution

Exact summation

How to calculate a RTN (rounded to nearest) sum faster.But we know how since 1970s.

Several algorithms have been proposed.FastSum (2006).AccSum (2008).FastAccSum (2008).iFastSum (2009).HybridSum (2009).OnlineExact (2010).


Solution

Error free transformation of two �oats

Algorithms to compute rounding errors.

TwoSum : requires 6 �op.

FastTwoSum : requires 3 �op but ordered summands.

TwoSum(A, B) = (S, E) such as A ⊕ B = S and A + B = S + E.


Solution

TwoSum and FastTwoSum


Solution

Exact summation

Given a vector of n �oating-point numbers with. We present somme exact summationalgorithms.


Solution AccSum

AccSum

Iterative algorithm.

Based on vector error free transformation.

Adapts automatically to the condition number of the sum.

Extract and sum the high order parts in each iteration.

Requires 4n �op for each itération.


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

AccSum


Solution AccSum

ExtractScalar

Figure 2.1 : ExtractScalar


Solution AccSum

ExtractScalar

Figure 2.1 : ExtractScalar


Solution AccSum

n cond Iterations

103 108 2

103 1024 3

106 1024 4

Table 2.1 : Number of iterations of the algorithm AccSum


Solution FastAccSum

FastAccSum

Improvement for AccSum.

FastAccSum requires only 3n �op for each iteration.theorically 25% faster.


Solution FastAccSum

AccSum


Solution FastAccSum

AccSum


Solution FastAccSum

AccSum


Solution FastAccSum

AccSum


Solution FastAccSum

AccSum


Solution FastAccSum

AccSum


Solution FastAccSum

AccSum


Solution FastAccSum

FastAccSum VS AccSum

Table 2.2 : Ratio of computing times AccSum / FastAccSum


Solution iFastSum

iFastSum

Pure distillation algorithm.

Delete zeros at the end of each iteration to reduce the size of vector.


Solution iFastSum

iFastSum


Solution iFastSum

iFastSum


Solution iFastSum

iFastSum


Solution iFastSum

iFastSum


Solution iFastSum

iFastSum


Solution HybridSum

HybridSum

Splits the summands so the standard �oating-point numbers can be considered as ahigh accumulators.

Accumulate the summands with the same exponent in an appropriate accumulator.

Use iFastSum to sum the intermediate accumulators.


Solution HybridSum

HybridSum


Solution HybridSum

HybridSum


Solution HybridSum

HybridSum


Solution OnlineExact

OnlineExact

Use the same idea of HybridSum but using two �oating-point numbers asaccumulator instead of spliting summands.



OnlineExact



OnlineExact



OnlineExact


Solution Compare algorithms

Hardware

Hardware

Two sockets.

Xeon E5 (2,2 Ghz, 8 cores).

Cache :

L1 : 32 KB.L2 : 256 KB.L3 : 20 MB Shared.

Memory max bandwidth 51,2 GB/s.

Turbo boost and multithreading are turned o�.



Compiler

Compiler

ICC -O3 -axCORE-AVX-I -fp-model double -fp-model strict

-funroll-all-loops

-axCORE-AVX-I : To indicate instruction set.

-fp-model double : Rounds intermediate results to 53-bit precision.

-fp-model strict : Disable contractions.

-funroll-all-loops : Unroll loops.



Compare algorithms for cond = 108



Compare algorithms for cond = 1032



Compare algorithms for de�rent condition numbers



Runtime of AccSum for entries with di�erent condition numbers



Runtime of FastAccSum for entries with di�erent condition numbers



Runtime of HybridSum for entries with di�erent condition numbers



Runtime of OnlineExact for entries with di�erent condition numbers


Optimization

Sommaire


2 Solution

3 OptimizationHybridSumOnlineExact

4 Parallelism



Optimization HybridSum

HybridSum

ALGORITHM HybridSum.INPUT : A, an array of floating point summands.OUTPUT : S, the correctly rounded sum of A.BEGIN.

1 Declare an intermediate array C.

2 FOREACH element of A as a do.

1 split(a, ah, al).2 i = exponent(ah).3 Ci += ah.4 i = exponent(al).5 Ci += al.

END FOREACH.

3 RETURN iFastSum(C).

END.



HybridSum



2 FOREACH 8 element of A as a do.

1 split(a, ah, al).2 i = exponent(ah).3 Ci += ah.4 i = exponent(al).5 Ci += al.

END FOREACH.


END.



HybridSum




1 prefetch data for the next loops.2 split(a, ah, al).3 i = exponent(ah).4 Ci += ah.5 i = exponent(al).6 Ci += al.

END FOREACH.


END.



HybridSum




1 prefetch data for the next loops.2 split(a, ah, al).3 i = exponent(ah).4 Ci += ah.5 i = i - 27.6 Ci += al.

END FOREACH.


END.



Progress in optimization of HybridSum



Progress in optimization of HybridSum


Optimization OnlineExact

OnlineExact

ALGORITHM OnlineExact.INPUT : A, an array of floating point summands.OUTPUT : S, the correctly rounded sum of A.BEGIN.

1 Declare two intermediate arrays C1, C2.

2 FOREACH element of A as a do.

1 i = exponent(a).2 (C1i, a) = 2Sum(C1i, a).3 C2i += a.

END FOREACH.

3 RETURN iFastSum(C1 ∪ C2).

END.



OnlineExact




1 i = exponent(a).2 (C1i, a) = 2Sum(C1i, a).3 C2i += a.

END FOREACH.


END.



OnlineExact




1 prefetch data for the next loops.2 i = exponent(a).3 (C1i, a) = 2Sum(C1i, a).4 C2i += a.

END FOREACH.


END.



OnlineExact


1 Declare an intermediate arrays C.


1 prefetch data for the next loops.2 i = exponent(a).3 (C2∗i, a) = 2Sum(C2∗i, a).4 C2∗i+1 += a

END FOREACH.


END.



Progress in optimization of OnlineExact



Progress in optimization of OnlineExact


Parallelism

Sommaire


2 Solution

3 Optimization

4 ParallelismImplementation and tests



Parallelism

Architecture


Parallelism

OpenMP


Parallelism

OpenMP


Parallelism

OpenMP


Parallelism

OpenMP


Parallelism

latency


Parallelism

bandwidth


Parallelism

MPI


Parallelism

MPI


Parallelism

MPI


Parallelism

MPI


Parallelism

Hybrid


Parallelism

Hybrid


Parallelism Implementation and tests

Parallel HybridSum



Parallel HybridSum



Parallel HybridSum



Parallel HybridSum



Parallel HybridSum



Parallel HybridSum



Parallel HybridSum



Parallel OnlineExact



HybridSum

Figure 4.1 : Scaling of HybridSum



OnlineExact

Figure 4.2 : Scaling of OnlineExact



Scaling

Algorithm 1 core 2 cores 4 cores 8 cores 16 cores

HybridSum (cycles) 249855884 141459028 72300156 37207844 19066140

OnlineExact (cycles) 259167668 156856764 91386036 46004832 23156420

Hybrid / seq 1 0,5661 0,2893 0,1489 0,0763

Online / seq 1 0,6052 0,3526 0,1775 0,0893

Online / Hybrid 1,0372 1,1088 1,2639 1,2364 1,2145

Table 4.1 : HybridSum vs OnlineExact



HybridSum weak scalingData size = 220 * number of cores



OnlineExact weak scalingData size = 220 * number of cores



Compare to other algorithms

Optimized sum.dasum : Optimized by Intel in the library MKL.

reproducible solutions.ReprodSum : Guarantee reproducibility of results (based on "AccSum").FastReprodSum : Faster than ReprodSum but requires direct rounding mode (basedon "FastAccSum").



ReprodSum

Figure 4.3 : How does it work ?



ReprodSum

Figure 4.3 : How does it work ?



Sequential results



Sequential results



Sequential results

1 FastReprodSum is 2,2times slower than dasum.

2 ReprodSum is 2,8 timesslower than dasum.

3 Hybrid and Online are 4times slower than dasum.

4 Hybrid and Online are 2times slower thanFastReprodSum.

5 Hybrid and Online are 1,5times slower thanReprodSum.



4 cores parallel results







1 The same ratios exceptfor OnlineExact.

2 Poor scaling ofOnlineExact.










1 HybridSum is as fast asReprodSum.

2 Due to limit of bandwidth.



ParallelismScaling

Algorithm 1 core 2 cores 4 cores 8 cores 16 cores

HybridSum 1 1.72 3.46 6.72 13.11

OnlineExact 1 1,66 2,84 5,63 11.20

FastReprodSum 1 1.92 3.46 6.51 8.68

ReprodSum 1 1,95 2,84 5,63 9.43

dasum 1 1.89 3.47 5.21 7.80

Table 4.2 : Scaling of algorithms relatively to number of cores



Scaling of ReprodSum



Scaling of FastReprodSum



Scaling of dasum


Conclusion and Work in progress

Sommaire


2 Solution

3 Optimization

4 Parallelism




Conclusion

More precision requires more computing time.

The Fastest algorithms are neither precise nor reproducibe.

We are trying to develop a reproducibe BLAS thar guarantees the best reportPrecision / Performance.



Work in progress

Tests on machines with more sockets and cores.

Generalize to dot.

Auto tuning.




Date post:	13-Jul-2015
Category:	Education
Upload:	chemseddine-chohra
View:	103 times
Download:	0 times

Presentation

Education