The Practice of Standards Formalization
Victor KuliaminInstitute for System Programming (ISP RAS)
Moscow, Russia
Outline• ISP RAS Background
– OS Testing – Compiler Testing– Protocol Testing– Hardware Testing
• ISP RAS Technologies (brief summary)• Linux Standard Base Support Activity
– LSB Infrastructure– LSB conformance testing– Detailed: Math conformance testing
April 21, 2023 Practice of Standards Formalization 2/28
ISP RAS Background• Operating System testing
– Regression test suite for Nortel Switch OS 1994-2000
– POSIX conformance test suite for real-time OS 2005-…– Linux Standard Base conformance test suites 2005-…– ARINC 653 2008-…
• Compiler testing– Part of C expressions dynamic semantics 2000-2003– Static semantics specification and test suites 2002-2004– Optimizing units testing in gcc and Intel compilers 2001-2003
• Protocol testing– Testing Microsoft Research IPv6 2000-2001– Test suite for Microsoft Mobile IPv6 2002-2003– Test suite for IPsec 2004-…
• Hardware testing– MIPS-based processors with DSP extensions 2006-…
April 21, 2023 Practice of Standards Formalization 3/28
ISP RAS Technologies• Model based testing
– KVEST (1996)• RSL specifications• Protel target language
– UniTESK (2001)• Specifications in extensions of target languages
(C – 2001, Java – 2001, C# – 2003)• Concurrency testing extension (2001)• Compiler front-end testing support (2004)• Hardware testing support (2006)• Combinational test generation (2007)• Math extension (2007)
• Static analysis
April 21, 2023 Practice of Standards Formalization 4/28
Oracle
State modelState model
Model Based Testing General Scheme
April 21, 2023 Practice of Standards Formalization 5/28
System under test
Behavior model
Test action generator
Coverage metric
12%
Coverage criteria
36%57%87%
Linux Standardization• What to do with 550 distributions of Linux?• Linux Standard Base
– Binary interface standard– Supported by Free Standards Group– Includes
• StandardsPOSIX, X/Open Curses, Open GL, Large File Support, …
• LibrarieslibXML, gtk, Qt, JDK, Perl, …
– 45000 functions in C• 2000 described accurately (POSIX)• 7000-9000 have good description of main functionality• Others have very poor or just no description
April 21, 2023 Practice of Standards Formalization 6/28
ISP RAS Activity for LSB Support
• LSB Infrastructure development– DB of distributions, libraries, profiles, operations, etc.– Conformance checking and certification of distributions
• Static analysis tools• Test suites
– Conformance checking of applications• Monitoring tools• Test suites
• LSB evolution support– Analysis DB and information system
• Linux driver verification
April 21, 2023 Practice of Standards Formalization 7/28
Test Development Levels
April 21, 2023 Practice of Standards Formalization 8/28
2000 accurately described– the only target for formalization– UniTESK conformance testing
7000-9000 partially well-defined – manual test development
specialized massive automated test construction technology
45000
UniTESK API Test Development Basics
• API partitioned into logical modules (classes)• Each module state is modeled• Module operations described with stateful contracts
– Preconditions (on state and operation parameters)– Postconditions (on pre-state, post-state, operation
parameters and results)– Invariants (on state, hold when no op is working)
• Structure of postcondition gives test coverage criteria• Coverage-targeted FSM abstraction for module• Testing – automatic on-the-fly exploration of FSM
April 21, 2023 Practice of Standards Formalization 9/28
Math Library Standards
• IEEE 754 (Floating-point arithmetics)FP numbers, basic operations
• ISO 9899 (C language and libraries)56 real + 16 complex functions
• IEEE 1003.1 (POSIX)63 real + 22 complex functions
• ISO 10697.1-3 (Language independent arithmetics)Elementary real and complex functions
April 21, 2023 Practice of Standards Formalization 10/28
IEEE 754 Floating-Point Numbers
April 21, 2023 Practice of Standards Formalization 11/28
• Normal : E > 0 & E < 2k –1 X = (–1)S·2(E–B)·(1+M/2(n–k–1))
• Denormal : E = 0 X = (–1)S·2(–B+1)·(M/2(n–k–1)) • Exceptional : E = 2k –1
– M = 0 : +, –– M ≠ 0 : NaN
sign
k+1 n-1
0
exponent mantissa
0 1 1 1 1 1 1 0 1 0 01 0 0 0 0 0 0 0 0 00 0 0 0 0 00 0 0 0
0 1 k
n, k
S E MB = 2(k–1) –1
2(–1)·1.1012 = 13/16 = 0,8125
0, -0
1/0 = +, (–1)/0 = –0/0 = NaN
n = 32, k = 8 – float (single precision)n = 64, k = 11 – doublen = 79, k = 15 – extended doublen = 128, k = 15 – quadruple
1/2(n-k-1) – 1 ulp
• Operations: +, –, *, /, sqrt , fma (2008), type conversions, remainder
• Correct rounding – 4 rounding modes– to +– to –– to 0– to the nearest
• NaN and infinite results• Exception flags
– INVALID : Incorrect arguments (NaN result)– DIVISION-BY-ZERO : Infinite result (precise ±∞)– OVERFLOW : Too big result (approximate ±∞)– UNDERFLOW : Too small (or denormal) result– INEXACT : Inexact result
IEEE 754 Computations
April 21, 2023 Practice of Standards Formalization 12/28
0
ISO C and POSIX Requirements
• ISO/IEC 9899 (C language) : 54 real functions – Exact values : sin(0) = 0, log(1) = 0, …– DIVISION-BY-ZERO flag : log(0), atanh(1), pow(0,x), Г(-n)– NaN results and INVALID flag outside of domains
• IEEE 1003.1 (POSIX) : 63 real + 22 complex– All IEEE 754 flags (except for INEXACT) for real functions– errno setting: Domain error, Range error – If x is denormal
f(x) = x for f(x)~x in 0 (sin, asin, sinh, expm1…)– In overflow HUGE_VAL should be returned
(value of HUGE_VAL unspecified)– Sometimes non-NaN results on NaN arguments
fmax(NaN, x) = x, pow(NaN, 0) = 1
April 21, 2023 Practice of Standards Formalization 13/28
Inconsistency with rounding modes
Source of non-interoperability
glibc : +∞MSVCRT : max double (1.797693134862316e+308)Solaris libc : max float (3.402823466385289e+38)
Inconsistency with IEEE 754
Example of POSIX RequirementsNAME sin, sinf, sinl - sine function SYNOPSIS #include <math.h> double sin(double x); float sinf(float x); long double sinl(long double x);DESCRIPTIONThese functions shall compute the sine of their argument x, measured in radians.An application wishing to check for error situations should set errno to zero and call feclearexcept(FE_ALL_EXCEPT) before calling
these functions. On return, if errno is non-zero or fetestexcept(FE_INVALID | FE_DIVBYZERO | FE_OVERFLOW | FE_UNDERFLOW) is non-zero, an error has occurred.
RETURN VALUEUpon successful completion, these functions shall return the sine of x.If x is NaN, a NaN shall be returned.If x is ±0, x shall be returned.If x is subnormal, a range error may occur and x should be returned.If x is ±Inf, a domain error shall occur, and either a NaN (if supported), or an implementation-defined value shall be returned. ERRORSThese functions shall fail if:Domain Error The x argument is ±Inf. If the integer expression (math_errhandling & MATH_ERRNO) is non-zero, then errno shall
be set to [EDOM]. If the integer expression (math_errhandling & MATH_ERREXCEPT) is non-zero, then the invalid floating-point exception shall be raised.
These functions may fail if:Range Error The value of x is subnormal If the integer expression (math_errhandling & MATH_ERRNO) is non-zero, then errno
shall be set to [ERANGE]. If the integer expression (math_errhandling & MATH_ERREXCEPT) is non-zero, then the underflow floating-point exception shall be raised.
April 21, 2023 Practice of Standards Formalization 14/28
ISO 10697 Requirements Real and complex elementary functions (no erf, gamma, j0, y1, … ) Only symmetric rounding modes (no rounding to + or to –)
• Preservation of sign• Preservation of monotonicity• Inaccuracy 0.5-2.0 ulp• Evenness and oddity• Exact values : cosh(0) = 1, log(1) = 0, …• Asymptotics near 0 : cos(x) ~ 1, sin(x) ~ x, …• Relations : expm1 <= exp, cosh >= sinh, atan <= ↓( π/2 ) , …
April 21, 2023 Practice of Standards Formalization 15/28
for sin, cos, tan – small arguments only
Summary of Requirements• Domain boundaries and poles (+ flags)• Exact values, limits and asymptotics• Preservation of sign and monotonicity• Symmetries
Evenness, periodicity, others : Г(1+x) = x·Г(x) • Relations and range boundaries• Precision
Correct rounding (according to mode) Computational accuracy Interoperability and portability of libraries and applications Feasible – ~ia64 (Intel), crlibm (INRIA)
April 21, 2023 Practice of Standards Formalization 16/28
Requirements TestedExtension of IEEE 754 on all library functions
• Correctly rounded results for 4 modes– Except for ones contradicting to range boundaries
• Infinite results in overflow and precise infinity cases– In overflow rounding to 0 returns the biggest finite number
• NaN results outside of function domain (and for NaN args)• Exception flags
– INVALID (and EDOM for errno) : Incorrect arguments– DIVISION-BY-ZERO (and ERANGE for errno) : Infinite result – OVERFLOW (and ERANGE for errno) : Too big result – UNDERFLOW (and ERANGE for errno) : Too small result ( + dnr)– INEXACT : Inexact result
April 21, 2023 Practice of Standards Formalization 17/28
Test Data Sources
• Bit structure of FP numbers– Boundaries
• 0, -0, +, -, NaN • Least and greatest positive and negative, normal and denormal
– Mantissa patternsFFFFFFFFFFFFF16 FFFFF1111000016 555550000FFFF16
Both arguments and values of a function
• Intervals of uniform function behavior
• Points hard to compute correctly rounded result
April 21, 2023 Practice of Standards Formalization 18/28
rint(262144.25)↑ = 2621440100000100010000000000000000000100000000000000000000000000000000x10000010001xxxxxxxxxxxxxxxxxx0100000000000000000000000000000000
Intervals and Boundaries• Neighbourhoods of 0, ±∞• Poles and overflow points• Zeroes and extremes• Tangents and asymtotics
– horizontal and diagonal
April 21, 2023 Practice of Standards Formalization 19/28
max
0
Table Maker Dilemma
April 21, 2023 Practice of Standards Formalization 20/28
tan(1.11011111111111111111111111111111111111111111000111112·2-22) = 1.1110000000000000000000000000000000000000000101010001 0 178 010…2·2-22
sin(1.11100000000000000000000000000000000000000111000010002·2-19) = 1.1101111111111111111111111111111111111100000010111000 067 11101…2·2-19
j1(1.10000000000000000000000000000000000000000000000000112·2-23) = 1.0111111111111111111111111111111111111111111111101000 094 11001…2·2-22
Rounding to the nearestf = x.xxxxxxxxxx|011111111...1xx...f = x.xxxxxxxxxx|100000000...0xx...
Rounding to 0, +, -f = x.xxxxxxxxxx|00000000...0xx...f = x.xxxxxxxxxx|11111111...1xx...
?!0,5 ulp
Number of Hard PointsProbabilistic evaluationUniform independent bits distribution• Total N = 2(n-k-1) values• ~N·2-m have m consecutive equal bits
Real data for sin on exponent -16
April 21, 2023 Practice of Standards Formalization 21/28
Eval. 0, +, - N
54 0.5 0 1
53 1 1 2
52 2 4 4
51 4 6 6
50 8 10 12
49 16 19 21
48 32 32 37
47 64 70 67
46 128 142 106
45 256 280 239
44 512 547 518
43 1024 1073 996
42 2048 2103 1985
41 4096 4187 4040
40 8192 8325 8142
3386417804515981120643892082331156599120239393299838035242121518428537554064774221620930267583474709602068045686026362989271814411863708499869721322715946622634302011697632972907922558892710830616034038541342154669787134871905353772776431251615694251273653 · π/2 = 1.0110101011000101101100100110001011001010000111111110 1 857 011…2·2849
sin(1.01101010110001011011001001100010110010100001111111112·2849) =1.11111111111111111111111111111111111111111111111111 1 690110…2·2-1
j = 15
sqrt(1.00100101011001010110010111001010110111001011111101002) =
1.0001001000001111100110011001111010011001001101110100 0 150 000…2
F(x) = f(x) – a·x – b = c1x2 + c2x3 + c3x4 + …
F(x) = c1(G(x) )2, G(x) = x + d1x2 + d2x3 +…
G(x) = y x = H(y), H is the reversed series
xm = H(sqrt(m/c12z)) F(xm) – a·xm – b = m/2z
Hard Points Calculation• Exhaustive search• Continued fractions (Kahan, 1983)• Dyadic method (Tang, 1989; Kahan, 1994)• Reduced search (Lefevre, 1997)• Lattice reduction (Gonnet, 2002; Stehle, Lefevre, Zimmermann, 2003)• Integer secants method (2007)
April 21, 2023 Practice of Standards Formalization 22/28
...2921
1
115
17
13π
Feasible only for single precision numbers
X ≈ N·π; X = M·2m; 2(n – k – 1) <= M < 2(n – k)
π ≈ (2m·M)/Nsqrt(N·2m) ≈ M + ½; 2(n-k-1) <= M, N < 2(n-k) 2(m+2)·N = (2·M + 1)2 – j (2·M + 1)2 = j (mod 2(m+2))
2–z
Test Suite Composition• Hard points
– double• Some hard points with ≥ 48 additional bits can be found in crlibm tests
http://lipforge.ens-lyon.fr/projects/crlibm• Calculated (some) hard points with ≥ 40 additional bits for
sqrt, cbrt, sin, asin, cos, acos, tan, atan, sinh, asinh, cosh, tanh, atanh, exp, log, exp2, expm1, log1p, erf, erfc, j0, j1
– float (single precision)• All hard points with ≥ 17 additional bits for sqrt, cbrt, exp, sin, cos
– extended double• All with ≥ 53 additional bits for sqrt, some for sin, exp
• Test suites developed– double : 58 real variable POSIX functions
• Correct values calculated by Maple and MPFR
April 21, 2023 Practice of Standards Formalization 23/28
sqrt exp sin atan lgamma j1
Boundary 20 20 20 20 20 20
Intrevals 106 1622 3674 4242 11680 24538
Patterns 141009 138451 331744 155008 121502 109036
Hard points 170170 28587 62342 95512 0 29436
Other 84820 0 4616 0 229 5664
Total 396125 168680 402396 254782 133431 168694
Tested Libraries
April 21, 2023 Practice of Standards Formalization 24/28
ID Processor arch Library OS
x86 i686 glibc 2.5 Linux Fedora
ia64 ia64 glibc 2.4 Linux Debian
x86_64 x86_64 glibc 2.3.4 Linux RHEL
s390 s390 glibc 2.4 Linux Debian
ppc64 ppc64 glibc 2.7 Linux Debian
ppc32 ppc32 glibc 2.3.5 Linux SLES
sparc UltraSparc III Solaris libc Solaris 10
VC8 x86_64 MS Visual C 2005 Windows XP
VC6 i686 MS Visual C 6.0 Windows XP
Test Results: Details
April 21, 2023 Practice of Standards Formalization 25/28
x86
ia64
x86_64
s390
ppc64
ppc32
sparc
VC6
VC8
x86
ia64
x86_64
s390
ppc64
ppc32
sparc
VC6
VC8
j1 y0 y1
log10
tgamma
log2
lgamma
log1p
j0
exp2
atan erf
expm1 log
erfc
fabs logb sqrt cbrt exp
sin cos tan asin acos
trunc
asinh
rint
acosh
nearby int
atanh
ceil
sinh
floor
cosh
round
tanh
rint(262144.25)↑ = 262144
Exact
1 ulp errors*
2-5 ulp errors
6-210 ulp errors
210-220 ulp errors
>220 ulp errors
Errors in exceptional cases
Errors for denormals
Completely buggy Unsupported
logb(2−1074) = −1022expm1(2.2250738585072e−308) = 5.421010862427522e−20
exp(−6.453852113757105e−02) = 2.255531908873594e+15
sinh(29.22104351584205) = −1.139998423128585e+12
cosh(627.9957549410666) = −1.453242606709252e+272sin(33.63133354799544) = 7.99995094799809616e+22
sin(− 1.793463141525662e−76) = 9.801714032956058e−2
acos(−1.0) = −3.141592653589794
cos(917.2279304172412) = −13.44757421002838
erf(3.296656889776298) = 8.035526204864467e+8
erfc(−5.179813474865007) = −3.419501182737284e+287
to nearestto –∞
to +∞to 0exp(553.8042397037792) = −1.710893968937284e+239
Implementations with Same Results
April 21, 2023 Practice of Standards Formalization 26/28
ceil floor round trunc rint nearby int fabs logb sqrt cbrt exp exp2 expm1 log log10 log2 log1p
x86
ia64
x86_64
s390
ppc64
ppc32
sparc
VC6
VC8
sinh cosh tanh asinh acosh atanh sin cos tan asin acos atan erf erfc tgamma lgamma j0 j1 y0 y1
x86
ia64
x86_64
s390
ppc64
ppc32
sparc
VC6
VC8
Unsupported
Conclusion
• Formalization can uncover numerous issues in mature industrial standards like POSIX (and more in implementations)
• But it may be not only ineffective but even impossible
April 21, 2023 Practice of Standards Formalization 27/28
Thank you!Questions?
[email protected]/~kuliamin
Institute for System Programming, Software Engineering Departmentwww.unitesk.comwww.linuxtesting.org
April 21, 2023 28/28Practice of Standards Formalization