45
1 Coverage Metrics Wenchao Li EECS 219C UC Berkeley
1
Co
ve
rag
e M
etr
ics
We
nch
ao
Li
EE
CS
21
9C
UC
Be
rke
ley
2
Outlin
e o
f th
e lectu
re
•W
hy d
o w
e n
eed c
overa
ge m
etr
ics?
•C
rite
ria for
a g
ood c
overa
ge m
etr
ic.
•D
iffe
rent appro
aches to d
efine c
overa
ge
metr
ics.
•D
iffe
rent ty
pes o
f covera
ge m
etr
ics.
3
A d
iffe
rent
kin
d o
f covera
ge
4
Why d
o w
e n
eed C
overa
ge
Metr
ics?
~re
q
ack
LT
L s
pecific
ation:
Assert
G (
req→
F a
ck);
Extr
em
e C
ase:
0specific
ation
!
Ok,
so s
uppose n
ow
we k
no
w
we n
ee
d m
ore
specific
ations,
but
do w
e k
no
w w
hat
specific
ations t
o w
rite
?
Ante
cede
nt
Failu
re
The o
ther
extr
em
e?
1 s
pec f
or
every
sta
te t
ransitio
n
A s
imp
le E
xam
ple
:V
acuity
Dete
ction
5
Why d
o w
e n
eed C
overa
ge
Metr
ics?
In g
enera
l:•
specs a
re n
ot
necessari
ly c
om
ple
te;
•nee
d t
o p
rom
pt/
assis
t hard
ware
/soft
ware
teste
rs;
•tr
adeoff b
etw
een c
ost of
pro
vid
ing c
overa
ge
and
perf
orm
ance/r
elia
bili
ty;
•shou
ld w
e h
ave a
sin
gle
covera
ge m
etr
ics o
r m
any
app
lication
-depe
nd
ent
covera
ge m
etr
ics?
•covera
ge m
etr
ics in s
imu
lation-b
ased v
erification →
covera
ge m
etr
ics in form
al verification.
6
What
are
th
e c
rite
ria f
or
a g
ood
covera
ge m
etr
ics?
•D
irect
co
rresp
on
den
ce w
ith
bu
gs.
Question 1
: checkin
g incom
ple
teness o
f specific
ations ≠
findin
g
redundancie
s in the s
yste
m?
•R
easo
nab
le c
om
pu
tati
on
al an
d h
um
an
eff
ort
to
:(a
) com
pute
the m
etr
ics;
(b)
inte
rpre
t covera
ge d
ata
and g
enera
te s
tim
uli
to e
xerc
ise
uncovere
d a
spects
;(c
) achie
ve h
igh c
overa
ge;
Question 2
: 100%
covera
ge =
com
ple
te d
esig
n?
(d)
min
imal m
odific
ation to v
alid
ation fra
mew
ork
.•
Kn
ow
led
ge o
f th
e d
esig
n r
eq
uir
ed
?–
Covera
ge M
etr
ics for
Bla
ckbox
Testing.
[M. W
. W
hale
n e
t. a
l. 2
006] [A
jitha
Raja
n2006]
•“O
bserv
ab
ilit
y”
–com
ple
te s
pecific
ations v
s. abstr
act
specific
ations.
7
Definin
g C
overa
ge
(1)
“Sim
ula
tion A
ppro
ach”:
[S. K
atz
et.
al. 1
999]
•A
CT
L S
afe
ty p
ropert
ies.
•A
well-
covere
d im
ple
menta
tion s
hould
clo
sely
resem
ble
the r
educed table
au
of its
specific
ation.
•H
ence, a fully
covere
d im
ple
menta
tion is
bis
imila
rto
the r
educed table
au
of its
specific
ation, i.e. has the s
am
e s
et of
behavio
rs a
s the s
pecific
ation.
Bis
imu
lati
on
is
very
str
ict!
Th
e r
ed
uced
tab
leau
can
b
e h
ug
e!
8
Definin
g C
overa
ge
(1)
“Sim
ula
tion A
ppro
ach”
cont.:
•F
our
crite
ria in c
om
paring a
n im
ple
menta
tion I
with the r
educed
table
au S
of th
e s
pecific
ation.
(a)
UnIm
ple
mente
dS
tart
Sta
te, w
hic
h c
onta
ins the s
et of sta
tes w
0’
in W
0’fo
r w
hic
h a
ll w
0W
0have w
0’
sim
(w0).
(b)
UnIm
ple
mente
dS
tate
, w
hic
h c
onta
ins the s
et of sta
tes w
’W
’fo
r w
hic
h a
ll w
W
have w
’sim
(w).
(c)
UnIm
ple
mente
dT
ransitio
ns, w
hic
h c
onta
ins the s
et of tr
ansitio
ns
<w
’, u
’> R
’fo
r w
hic
h S
sim
ula
tes I
even w
ithout th
e tra
nsitio
n
<w
’, u
’>.
(d)
ManyT
oO
ne, w
hic
h c
onta
ins the s
et of sta
tes w
’W
’fo
r w
hic
h
sim
-1(w
’) is n
ot a s
ingle
ton.
∈∉
∈
∈∉
∈
∈
Four
crite
ria a
re e
mpty
iff
the im
ple
menta
tion a
nd the
reduced table
au
of th
e s
pecific
ation a
re b
isim
ilar.
9
Definin
g C
overa
ge
(2)
“Mu
tan
t-b
ase
d A
pp
roa
ch
”: [Y
. H
oskote
et.
al. 1
999]
•In
sp
ire
d b
y m
uta
tio
n c
ove
rag
e in
sim
ula
tio
n-
ba
se
d v
eri
fica
tio
n.
[D.
L.
Dill
1998
]
•F
orm
ally
, fo
r a
n im
ple
me
nta
tio
n I
(mo
de
led
as a
lab
ele
d s
tate
-tra
nsitio
n g
rap
h),
a s
tate
win
I,
an
d
an
ob
se
rva
ble
sig
na
l q
, w
e s
ay t
ha
t w
is q
-
co
ve
red
by a
sp
ecific
atio
n S
if I
w,q
’(m
uta
nt
imp
lem
en
tatio
n b
y flip
pin
g t
he
va
lue
of
qin
w)
do
es n
ot sa
tisfy
S.
10
Ho
w a
re t
hese 2
ap
pro
ach
es
diffe
rent?
qA
GA
Gq
¬∨
q~
~q
Syste
m I
1:
Sp
ecif
icati
on
:
Red
uced
tab
leau
S1:
Sim
ula
tio
n a
pp
roach
:all 4
cri
teri
a a
re e
mp
ty →
full c
overa
ge.
Mu
tan
t-b
ased
ap
pro
ach
:b
oth
sta
tes o
f I 1
are
no
t q
-co
vere
d.
11
Ho
w a
re t
hese 2
ap
pro
ach
es
diffe
rent?
Sp
ecif
icati
on
:A
Gq
Syste
m I
2:
Red
uced
tab
leau
S2:
q
Sim
ula
tio
n A
pp
roach
:
Cri
teri
a 4
is n
ot
em
pty
: b
oth
sta
tes o
f I 2
are
sim
ula
ted
by t
he s
tate
t0.
Mu
tan
t-b
ased
Ap
pro
ach
:
I 2is
q-c
overe
d b
y S
2.
u0
u1
t 0
12
Muta
nt-
based A
ppro
ach
Tw
o c
overa
ge c
hecks:
(1)
Fals
ity c
overa
ge: does the m
uta
nt F
SM
still
satisfy
the s
pecific
ation?
(2)
Vacuity c
overa
ge: if the m
uta
nt F
SM
still
satisfies the s
pecific
ation, does it satisfy
it v
acuously
?
13
w0
w4
gra
nt 2
w3
w1
gra
nt 1
w2
gra
nt 2
LT
L s
pecific
ation S
: assert
G (
gra
nt 1→
F g
rant 2
);
I w
ant an e
xam
ple
!
w4
is fals
ity-
covere
d b
y S
w.r
.t.
muta
ton
on g
ran
t2
w1
is v
acuity-
covere
d b
y S
w.r
.t.
om
issio
n o
f w
1o
r
muta
tion o
n g
ran
t 1
w0
is v
acuity-
covere
d b
y S
w.r
.t.
muta
tion o
n g
ran
t 2
?
(1)
G (
gra
nt 1
→X
gra
nt 2
)?
(2)
Num
ber
of g
ran
t 2=
2?
(3)
Redundancy,
i.e.
w2
can
be o
mitte
d?
str
uctu
re-c
overe
d
(flip
ped a
lwa
ys)
vs.
node-c
overe
d
(flip
ped o
nly
once)
Qu
esti
on
: Is
w4
str
uctu
re-
co
ve
red
or
no
de-c
ove
red
by S
w.r
.t.
the m
uta
tio
n o
n g
ran
t 2?
14
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
ased
co
ve
rag
e
•C
ircu
it c
overa
ge
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
vera
ge
•A
ssert
ion
co
ve
rag
e
•M
uta
tio
n c
overa
ge
15
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
as
ed
co
ve
rag
e
•C
ircu
it c
overa
ge
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
vera
ge
•A
ssert
ion
co
ve
rag
e
•M
uta
tio
n c
overa
ge
16
Code C
overa
ge (
sim
ula
tion)
•sta
tem
en
t co
ve
rag
e
•b
ran
ch
co
ve
rag
e
•e
xp
ressio
n c
ove
rag
e
•G
ive
n a
CF
G c
alle
d G
, fo
r a
n in
pu
t se
qu
en
ce
t(2
I )*
su
ch t
ha
t th
e e
xe
cu
tio
n o
f G
on
t,
pro
jecte
d
on
th
e s
eq
ue
nce
of
loca
tio
ns,
is l
0,…
,lm
, w
e s
ay
tha
t a
sta
tem
en
t s
is c
ove
red
by t
if t
here
is 0
≤j
≤m
s.t.
l jco
rre
sp
on
ds t
o s
; a
bra
nch
<l, l’>
is
co
ve
red
by t
if th
ere
is 0
≤j ≤
m-1
su
ch
th
at
l j=
la
nd
lj+
1=
l’.
∈
Wh
at
if t
here
is
co
ncu
rren
cy?
17
Code C
overa
ge (
F.V
.)
•G
iven a
CF
G c
alle
d G
and ξ
a
specific
ation s
atisfied in G
, w
e s
ay a
sta
tem
ent s
of G
is c
overe
d b
y ξ
if o
mitting
sfr
om
Gcauses v
acuous s
atisfa
ction o
f ξ
in the m
uta
nt C
FG
. S
imila
rly, a b
ranch
<l,l’>
of G
is c
overe
d if om
itting it
causes
vacuous s
atisfa
ction o
f ξ.
•W
hy v
acuous s
atisfa
ction
only
?
18
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
ased
co
ve
rag
e
•C
irc
uit
co
ve
rag
e
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
vera
ge
•A
ssert
ion
co
ve
rag
e
•M
uta
tio
n c
overa
ge
19
Circuit C
overa
ge
(sim
ula
tion)
•la
tch c
overa
ge
•to
ggle
covera
ge
•A
latc
h is c
overe
d if it c
hanges its
valu
e a
t le
ast once d
uring the e
xecution o
f th
e
input sequence.
•A
n o
utp
ut variable
is c
overe
d if its v
alu
e
has b
een toggle
d (
requires the v
alu
e to b
e
changed a
t le
ast tw
ice).
20
Circuit C
overa
ge
(F
.V.)
•R
ep
lace
th
e q
ue
stio
n b
y t
he q
ue
stio
n o
f w
he
the
r
dis
ab
ling
th
e c
ha
ng
e c
au
se
s t
he
sp
ecific
atio
n to
be
sa
tisfie
d v
acu
ou
sly
.
•A
la
tch
lis
co
ve
red
if th
e s
pecific
atio
n is
va
cu
ou
sly
sa
tisfied
in
th
e c
ircu
it o
bta
ine
d b
y
fixin
g t
he
va
lue
of
lto
its
in
itia
l va
lue
.
•A
n o
utp
ut
ois
co
vere
d if
the
sp
ecific
atio
n is
va
cu
ou
sly
sa
tisfied
in
th
e c
ircu
it o
bta
ine
d b
y
allo
win
g o
to c
ha
ng
e its
va
lue
on
ly o
nce
.
21
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
ased
co
ve
rag
e
•C
ircu
it c
overa
ge
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
vera
ge
•A
ssert
ion
co
ve
rag
e
•M
uta
tio
n c
overa
ge
22
Hit C
ount (s
imula
tion)
•R
epla
ce b
inary
covera
ge q
ueries w
ith
quantita
tive m
easure
ments
–th
e n
um
ber
of tim
es a
n o
bje
ct has b
een v
isited.
•V
isited o
ften →
functionalit
y b
etter
covere
d.
23
Hit C
ount (F
.V.)
•T
he m
inim
alnum
ber
of vis
its in w
hic
h w
e
have to p
erf
orm
the m
uta
tion (
or
om
issio
n
of th
e e
lem
ent)
in o
rder
to fals
ify the
specific
ation in the d
esig
n o
r to
make it
vacuously
satisfied.
24
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
ased
co
ve
rag
e
•C
ircu
it c
overa
ge
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
ve
rag
e
•A
ssert
ion
co
ve
rag
e
•M
uta
tio
n c
overa
ge
25
FS
M C
overa
ge (
sim
ula
tion)
•A
sta
te o
r a tra
nsitio
n o
f th
e F
SM
is
covere
d if it is v
isited d
uring the e
xecution
of th
e input sequence.
•T
ransitio
n c
overa
ge c
an b
e e
xte
nded to
path
covera
ge.
•P
roble
m?
lin
kin
g t
he u
nco
ve
red
part
s o
f th
e F
SM
to
u
nco
vere
d p
art
s o
f th
e
HD
L p
rog
ram
is n
ot
triv
ial!
co
mp
uti
ng
th
e p
ath
co
ve
rag
e i
s
exp
en
siv
e!
26
FS
M c
overa
ge (
F.V
.)
•In
sta
te c
overa
ge, w
e c
heck the influence
of om
issio
n o
f a s
tate
wor
changin
g the
valu
es o
f th
e o
utp
ut variable
s in w
on the
(nonvacuous)
satisfa
ction o
f th
e
specific
ation.
•In
path
covera
ge, w
e c
heck the influence
of om
itting o
r m
uta
ting a
fin
ite p
ath
on the
(nonvacuous)
satisfa
ction o
f th
e
specific
ation.
27
I w
ant an e
xam
ple
again
!
w0
w4
gra
nt 2
w3
w1
gra
nt 1
w2
gra
nt 2
LT
L s
pecific
ation S
: assert
G (
gra
nt 1→
F g
rant 2
);
28
I w
ant an e
xam
ple
again
!
w0
w4
gra
nt 2
w3
w1
gra
nt 1
w2
gra
nt 2
LT
L s
pecific
ation S
: assert
G (
gra
nt 1→
F g
rant 2
);
om
itti
ng
path
w0,
w2, w
3.
≠re
mo
vin
g
tran
sit
ion
s
<w
0,w
2>
an
d
<w
2,w
3>
Th
e s
pecif
icati
on
is s
ati
sfi
ed
n
on
vacu
ou
sly
.
29
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
ased
co
ve
rag
e
•C
ircu
it c
overa
ge
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
vera
ge
•A
ss
ert
ion
co
ve
rag
e
•M
uta
tio
n c
overa
ge
30
Assert
ion C
overa
ge (
sim
ula
tion)
•A
lso c
alle
d “
functional covera
ge”.
•A
ssert
ions c
an b
e p
ropositio
nalor
tem
pora
l.
•A
test tcovers
an a
ssert
ion a
if the
execution o
f th
e d
esig
n o
n t
satisfies a
.
•M
easure
s %
assert
ions c
overe
d for
a
giv
en s
et of in
put sequences.
31
Assert
ion C
overa
ge (
F.V
.)
•A
n a
ssert
ion a
is c
overe
d b
y a
specific
ation ξ
in a
FS
M F
if the m
uta
nt
FS
M F
’obta
ined fro
m F
by o
mitting a
ll behavio
rs that satisfy
asatisfies ξ
nonvacuously
.
•W
hat covera
ge m
etr
ic that w
e h
ave talk
ed
about is
sim
ilar
to this
one?
FS
M P
ath
Covera
ge!
32
Types o
f covera
ge m
etr
ics
A.S
ynta
ctic-c
overa
ge m
etr
ics
•C
od
e-b
ased
co
ve
rag
e
•C
ircu
it c
overa
ge
•H
it c
ou
nt
B.S
em
antic-c
overa
ge m
etr
ics
•F
SM
co
vera
ge
•A
ssert
ion
co
ve
rag
e
•M
uta
tio
n c
ov
era
ge
33
Muta
tion C
overa
ge (
sim
ula
tion)
•U
ser
intr
oduces a
sm
all
change to the
desig
n a
nd c
heck for
err
oneous b
ehavio
r.
•T
he c
overa
ge o
f a test tis
measure
d a
s
the %
muta
nt desig
ns that fa
il on t.
•T
he g
oal is
to fin
d a
set of in
put
sequences s
.t. fo
r each m
uta
nt desig
n
there
exis
ts a
t le
ast one test th
at fa
ils it.
34
Muta
tion C
overa
ge (
F.V
.)
•B
y m
uta
tion, w
e a
ctu
ally
mean local
muta
tion.
•W
e h
ave talk
ed a
bout th
is.
35
Muta
nt-
based A
ppro
ach
Tw
o c
overa
ge c
hecks:
[H.
Chockle
ret. a
l. 2
006]
(1)
Fals
ity c
overa
ge: does the m
uta
nt F
SM
still
satisfy
the s
pecific
ation?
(2)
Vacuity c
overa
ge: if the m
uta
nt F
SM
still
satisfies the s
pecific
ation, does it satisfy
it v
acuously
?
36
Com
puting C
overa
ge (
1)
•M
uta
nt-
based
Appro
ach:
[Y. H
osko
teet. a
l.
199
9]
•T
he g
oal is
to fin
d the s
et of covere
d
sta
tes.
•C
overa
ge =
num
ber
of covere
d s
tate
s /
num
ber
of re
achable
sta
tes
•R
ecurs
ive
alg
orith
m for
a g
iven A
CT
L
form
ula
and a
giv
en o
bserv
ed s
ignal.
37
Outlin
e o
f th
e a
lgorith
m
•S
ay,
we
wa
nt
to c
om
pu
te c
ove
rag
e f
or
A(f
1U
f 2).
)(
)(
''
21
00
fT
fT
SS
II
=f 1
S0
f 2
f 1
f 1
f 1
f 1
f 2
f 1f 1
f 2
C(S
0, A
(f1
Uf 2
)) =
C(t
ravers
e(S
0,f
1,f
2)
U C
(fir
str
each
ed
(S0,f
2),
f2)
)(
)(
''
21
00
fT
fT
SS
¬=
II
forw
ard
(S0)
giv
es s
tate
s
reach
ab
le i
n e
xactl
y o
ne s
tep
fr
om
th
e s
tart
sta
tes i
n S
0.
)))
,(
((
))(
(
),
(
22
0
20
20
ff
TS
forw
ard
edfi
rstr
eachf
TS
fS
edfi
rstr
each
¬
=
I
UI
)2
,1),'
0(
('
0
)2
,1,
0(
ff
Sfo
rwa
rdtr
avers
eS
ff
Str
avers
e
U
=
whe
re,
T(b
) re
pre
sen
t th
e
set
of
sta
tes w
hic
h
sati
sfy
b.
38
Com
puting C
overa
ge (
2)
modu
le e
xam
ple
(o1,o
2,o
3);
reg
o1,o
2,o
3;
initia
l beg
in
o1=
o2=
o3=
0;
end
alw
ays @
(posedg
eclk
) beg
in
assig
n o
1=
o1;
assig
n o
2=
o2|o
3;
assig
n o
3=
~o
3;
end
endm
odu
le
[H.
Ch
ockle
ret.
al.
2006]
S:
assert
G(o
2→
F o
3);
Co
de C
ov
era
ge:
This
sta
tem
ent
is
uncovere
d b
y t
he
specific
atio
n w
.r.t. to
both
om
issio
n a
nd m
uta
tions.
39
Com
puting C
overa
ge (
2)
modu
le e
xam
ple
(o1,o
2,o
3);
reg
o1,o
2,o
3;
initia
l beg
in
o1=
o2=
o3=
0;
end
alw
ays @
(posedg
eclk
) beg
in
assig
n o
1=
o1;
assig
n o
2=
o2|o
3;
assig
n o
3=
~o
3;
end
endm
odu
le
[H.
Ch
ockle
ret.
al.
2006]
S:
assert
G(o
2→
F o
3);
Cir
cu
it C
overa
ge:
Latc
h o
1is
uncovere
d
–fixin
g o
1to
0 f
or
the
wh
ole
execution d
oes
not
affect th
e
satisfa
ction
of S
.
40
Com
puting C
overa
ge (
2)
[H.
Ch
ockle
ret.
al.
2006]
S:
assert
G(o
2→
F o
3);
000
100
101
010
011
111
110
001
o1o
2o
3
FS
M C
ove
rag
e:
All
sta
tes in w
hic
h o
1 =
1 a
re u
nre
acha
ble
, and
thus u
ncovere
d w
.r.t. all
specific
atio
ns.
?
41
Com
puting C
overa
ge (
2)
[H.
Ch
ockle
ret.
al.
2006]
(1)
Naiv
e w
ay:
en
um
era
teth
rou
gh
all
mu
tan
t
FS
M t
o c
heck b
oth
fals
ity a
nd
vacu
ity c
overa
ge
for
a s
pecif
icati
on
ξ.
(2)
A B
ett
er
way:
co
mp
ute
co
ve
rag
e
sym
bo
licall
y.
oT
he id
ea i
s t
o l
oo
k f
or
a f
air
path
in
th
e
pro
du
ct
of
the m
uta
nt
FS
M F
’an
d a
n
au
tom
ato
n A
~ξ
for
the n
eg
ati
on
of ξ.
oA
dd
a n
ew
va
riab
le x
th
at
en
co
des a
su
bfo
rmu
lain
ξth
at
is b
ein
g r
ep
laced
by
tru
e/f
als
e.
Th
e v
alu
e 0
fo
r x s
tan
ds f
or
“n
o
rep
lacem
en
t”.
Th
en
we c
heck t
he
sati
sfa
cti
on
of ξ
in t
he s
yste
m.
oC
on
sid
er
an
au
gm
en
ted
pro
du
ct
wit
h s
tate
sp
ace 2
XX
2X
X S
.
…
…
A c
yc
le i
s r
ea
ch
ab
le in
th
e
au
gm
en
ted
au
tom
ato
n.
<w
,u0,s
0>
<w
,u,s
>
P
<w
,w’,s
><
w,w
,s>
∈
fair
pa
th
flip
pin
g
q
42
Co
mple
xity o
f C
om
puting C
ove
rage
3n
+ 2
m+
log
lH
it C
ount
3n
+ 4
m+
logk
Assert
ion C
overa
ge
3n
+ 4
m+
logp
FS
M P
ath
Covera
ge
2n
+ 3
m+
log
lC
ircuit C
overa
ge
2n
+ 2
m+
logk
Co
de C
ove
rage
3n
+ 3
mV
acu
ity C
overa
ge
3n
+ 2
mM
uta
tion C
overa
ge
Co
mp
lexit
yM
etr
ic
Co
mp
lexit
y i
n t
erm
s o
f R
OB
DD
vari
ab
les r
eq
uir
ed
.
nand m
are
the
num
ber
of
variable
s r
equired fo
r e
ncodin
g the s
tate
space o
f F
and A
~ξ;
ldenote
s the n
um
ber
of
latc
hes in the c
ircuit, k
de
note
s the n
um
be
r o
f
assert
ions/n
um
ber
of lin
es o
f nodes; p
denote
s th
e length
of
the p
ath
; tdeno
tes th
e
thre
shold
of hit c
ount.
43
Conclu
sio
n
•T
wo p
roble
ms inhere
nt w
ith a
ny c
overa
ge
metr
ics.
•T
wo a
ppro
aches to d
efine c
overa
ge
metr
ics for
form
al verification.
•D
iffe
rent sem
antic a
nd s
ynta
ctic c
overa
ge
metr
ics –
how
are
they s
imila
r to
/diffe
rent
from
the o
nes u
sed in s
imula
tion.
•S
till
an o
pen p
roble
m.
44
Refe
rence L
ist
•M
ichae
l W
. W
hale
n,
Ajit
ha
Raja
n,
Mats
P.E
. H
eim
dahl, S
teven P
. M
iller.
C
overa
ge m
etr
ics f
or
require
ments
-base
d t
esting.
Pro
c.
of
Inte
rnationa
l S
ym
posiu
m o
n S
oft
ware
Testing a
nd A
naly
sis
, page 2
5-3
6,
2006.
•A
jitha
Ra
jan.
Covera
ge M
etr
ics t
o M
easure
Adequ
acy o
f B
lack-B
ox T
est
Suites.
Pro
c.
of
Inte
rnationa
l C
onfe
rence o
n A
uto
mate
d S
oft
wa
re
Engin
eeri
ng,
page 3
35-3
38,
2006.
•S
. K
atz
, O
. G
rum
berg
, D
. G
eis
t. “
Have I
wri
tten e
nou
gh p
rop
ert
ies? -
A
meth
od o
f com
pariso
n b
etw
ee
n s
pecific
ation a
nd im
ple
me
nta
tion”.
Pro
c.
of
10th
AC
M A
dvanced R
esearc
h W
ork
ing C
on
fere
nce o
n C
orr
ect
Hard
ware
D
esig
n a
nd V
erification M
eth
ods (
CH
AR
M),
page 2
80-2
97,
1999.
•D
.L.
Dill
. W
hat’s b
etw
een s
imula
tio
n a
nd f
orm
al verification
?P
roc.
of
35th
D
esig
n A
uto
mation C
onfe
ren
ce,
page 3
28–
329,
199
8.
•Y
. H
oskote
, T
. K
am
, P
.-H
Ho, X
. Z
hao.
Covera
ge e
stim
ation f
or
sym
bo
lic
model ch
eckin
g.
Pro
c.
of
36th
Desig
n A
uto
mation C
onfe
ren
ce,
page 3
00–
305,
199
9.
•H
ana
Chockle
r, O
rna
Kupfe
rman,
Moshe Y
. V
ard
i. C
overa
ge m
etr
ics f
or
form
al verification.
Inte
rnal Jo
urn
al o
n S
oft
ware
Tools
for
Techno
log
y
Tra
nsfe
r (S
TT
T),
Vol. 8
, Is
sue 4
, P
age 3
73-3
86,
August
20
06.
45
Refe
rence L
ist
•H
ana
Chockle
r, O
rna
Kupfe
rman,
Moshe Y
. V
ard
i. C
overa
ge m
etr
ics f
or
form
al verification.
Pro
c.
of
12th
Advance
d R
esearc
h W
ork
ing C
onfe
rence
on C
orr
ect
Hard
ware
Desig
n a
nd V
erificatio
n M
eth
ods (
CH
AR
ME
), 2
003.
•H
ana
Chockle
r, O
rna
Kupfe
rman.
Covera
ge o
f Im
ple
menta
tions b
y
Sim
ula
ting S
pe
cific
ations.
Pro
c.
Of 2nd I
FIP
Inte
rnatio
nal C
Onfe
rence
on
Theore
tical C
om
pute
r S
cie
nce:
Foundatio
ns o
f In
form
ation T
echnolo
gy in
the E
ra o
f N
etw
ork
ing a
nd M
obile
Com
putin
g,
page 4
09-4
21,
2002.
•H
ana
Chockle
r, O
rna
Kupfe
rman,
Robert
P.
Kurs
han,
Moshe Y
. V
ard
i. A
P
ractical A
ppro
ach t
o C
overa
ge in M
ode
l C
he
ckin
g.
In P
roc.
of
13th
In
tern
ation
al C
onfe
rence o
n C
om
pute
r A
ide
d V
erificatio
n,
page 6
6-7
8,
2001.
•H
ana
Chockle
r, O
rna
kupfe
rman,
Moshe Y
. V
ard
i. C
overa
ge M
etr
ics f
or
Tem
pora
l Logic
Model C
heckin
g.
In P
roc.
of 7th
Inte
rnation
al C
onfe
rence
on T
ools
and A
lgorith
ms f
or
the C
onstr
uctio
n a
nd A
na
lysis
of
Syste
ms,
page 5
28-5
42,
2001.
•O
rna
Kupfe
rman,
Moshe Y
. V
ard
i. V
acuity D
ete
ction in T
em
pora
l M
odel
Checkin
g.
In P
roc.
of
10th
IF
IP W
G 1
0.5
Advanced R
esearc
h W
ork
ing
Confe
rence o
n C
orr
ect
Hard
ware
Desig
n a
nd
Verification M
eth
ods,
pag
e
82-9
6,
19
99.
