APost-ProcessingAlgorithmAppliedtoReduce

theSizeoftheCoveringArraysoftheNIST

Repository

Ph.D.Jose

Torres-Jimenez

LaboratoriodeTecnologıasdeInform

acion,CINVESTAV-Tamaulipaskm

5.5

Carretera

Cd.Victoria-Soto

laMarina,87130,Cd.VictoriaTamps.,Mexico.

September

19,2013

1/72

Contents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

2/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

3/72

Introduction

SoftwareTesting

�Now

adayssoftwareproductsarepresentin

almostallhuman

activities,then

itisnecessary

toassure

ahighlevelof

function-

alityof

thesoftwareproducts.

�Itisestimated

that

atleast50%

ofthecost

ofdevelopinga

new

softwarecomponentisrelatedto

thetestingprocess.1

1B.HailpernyP.Santhanam

,2002[1].

A.Hartm

an,2005[2].

4/72

Introduction

SoftwareTesting

�Oneoption

totestasoftwarecomponentisto

usean

exhaustive

approach,i.e.

test

allthepossible

combinationsof

theinput

param

eters.

�Another

option

isto

use

combinatorialtesting,

that

guarentees

that

allthecombinationsof

certainnumber

ofparam

etersis

tested

exactlyor

atleastcertainnumber

oftimes.2

2Cohen

etal.[3]yCohen

etal.[4].

5/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

6/72

BasicDefinitions

Orthogonal

Arrays(OA)

Definition

Let

beN,t,

k,v,andλfive

positive

integers,an

orthogonalarray

OAλ(N

;t,k

,v)isan

N×karrayA=(a

i,j),0≤

i≤

N−1,

0≤

j≤

k−1,

over

Zv=

{0,1,...,v

−1}

withtheproperty

that

foranytdistinct

columns,thereareexactlyλrowsthat

takeseach

valueof

Zt v=

{0,1,...,v

−1}

t.

�When

λ=1theOAisof

unitaryindex.

�For

OAsof

unitaryindex

andvaprimepow

erthereexists

anoptimalsolution

3

3Bush

[5]

7/72

00

00

00

01

00

11

01

01

01

01

10

01

01

11

10

11

01

01

10

01

10

01

11

10

01

11

00

01

Figure

1:TheorthogonalarrayOA2(8;2,7,2).

8/72

BasicDefinitions

Orthogonal

Arrays(OA)

�Thecase

when

visaprimepow

erandv≥

t,issolved

using

arithmeticof

aGaloisFiniteField.Thevalueof

each

cellof

theOAistheevaluationof

thenumber

ofrowin

thenumber

ofcolumn,thelast

columnisdirectlytheleftmostcoeffi

cient

ofthenumber

ofrow.Thisprocedure

canbeimplemented

usingalogaritm

tableforaGaloisFiniteField

4

�Thecase

when

visaprimepow

erandv<

tissolved

using

thezerosum

algorithm.

4Torreset

al.[6]andTorreset

al.[7]

9/72

BasicDefinitions

CoveringArrays(CA)

TheOAsareagoodoption

totesteffi

cientlysoftwarecomponents,

butthey

havethedrawbackthat

they

donot

existforcertaincom-

binationsof

param

eters(λ,k

,v,t).

Arelaxedversionof

theOAs

aretheCAs(eachcombinationmust

existat

leas

once).

Given

the

relaxation,they

existforanycombinationof

param

etersandthey

havealower

number

ofrows.

Definition

Let

beN,t,

k,andvfourpositiveintegers,acoveringarray

CA(N

;t,k

,v)5

isan

N×karrayA=(a

i,j),0≤

i≤

N−1,

0≤

j≤

k−1,

over

Zv=

{0,1,...,v

−1}

withtheproperty

that

foranytdistinct

columns,at

leastonerowtakeseach

valueof

Zt v=

{0,1,...,v

−1}

t.

5when

thecolumnshavedifferentorder

itiscalledaMixed

CoveringArray

(MCA)

10/72

BasicDefinitions

CoveringArray

Number(C

AN)

Thecoveringarrayconstructionproblem

(CACP)consistsin

con-

structingacoveringarrayCA(N

;t,k

,v)giventheparam

eterst,

k,

andvin

such

away

thenumber

ofrowsN

ofthecoveringarray

isminimal.Thesm

allest

Nforwhichacoveringarrayexists

isthe

coveringarraynumber

(CAN)fortheparam

eterst,

k,andv,and

itisdenoted

by CAN(t,k

,v)=min{N

|∃CA(N

;t,k

,v)}.

(1)

11/72

BasicDefinitions

NP-Complete

andSearchSpaceforCAs

�Even,thereisnoproof

that

CACPbelongs

totheclassof

NP-

Complete

problems,somerelatedproblemsareNP-Com

plete.

For

instance:theproof

ifitispossibleto

addanew

rowthat

provides

atleastacertainnumber

oft-wisemissingcombina-

tions6

�Thesearch

spaceof

theCACPdenoted

bySsatisfies:

vt≤

S≤

�vk N

�

(2)

6Colbourn

[8]

12/72

BasicDefinitions

Optimal

CoveringArrays

Therearereportedonlyasm

allnumber

ofoptimalcoveringarrays:

�Thecase

CA(v

t;t,m

ax(v,t)+1,v)when

visaprimepow

er7.TheCAsconstructed

inthisway

areequivalentto

OAsof

unitaryindex.

�Thecase

(v=2)

∧(t

=2)

wherek≤

� N−1

�N 2�

�8

Ingeneralthedeterminationof

theCANisvery

difficultandisthe

them

eofalotof

research,butasym

ptotically

(for

largek)

CAN(t,k

,v)≈

vtlog(k)

(3)

7Bush

[5]

8Renyi

[9],Katona[10],Kleitman

andSpencer[11]

13/72

BasicDefinitions

Isomorphism

inCAs

Given

that:

�Thepositionof

therowsin

aCAisnot

relevant,alltheCAs

obtained

bypermutingtherowsof

aCAareisom

orphic.

�Ifallthecolumnsin

aCA

havethesameorder

(alphabet),

alltheCAsobtained

bypermutingthecolumnsof

aCA

are

isomorphic.

�Thecoverage

properties

ofaCAarenot

affectedifthesym-

bolsin

onecolumnarepermuted(for

instance

allzerosare

exchangedwithallones),

alltheCAsobtained

bypermuting

symbolswithin

columnsareisom

orphic.

Definition

For

aCAthereareN!k!(v!)kisom

orphicCAs.

N!permutation

ofrows,k!

permutationsof

columns,and(v!)kpermutationsof

symbolsin

thecolumns.

14/72

Given

twoCAsAandBCA(6;2,5,2)they

areisom

orphicifonecan

beconstructed

from

theotherusingsomepermutation

ofrows,per-

muationof

columns,andpermutation

ofsymbols.

Consideringthat

τdefines

therow

permutation,πisthecolumnpermutation,and

φdefines

thesymbol

permutation.In

thenextfigure

weillustrate

how

toconstruct

Bfrom

A.

A=

11

10

11

01

10

00

00

11

10

10

01

10

00

11

11

B=

01

10

10

01

00

11

01

01

00

01

00

01

11

01

11

τ=(0

12345)

π=(0

1234)

φ=(0

0000)

τ�=(3

12540)

π�=(4

2031)

φ�=(0

1010)

Figure

2:UsingCAAweobtain

CAB

usingτ� ,π� ,andφ� .

15/72

BasicDefinitions

Redundancy

inCAs

ThetwocasesofoptimalCAsmentioned

previosuly(O

Aswith(λ

=1)∧(primepow

er(v)),and(v

=2)

∧(t

=2))resultin

CAsthat

do

not

haveredundancy.For

anyother

case

thepossibility

that

aCA

has

alotof

redundancy

isvery

high.Aredundantelem

ent(usually

denotedas

wildcard)cantake

anyvalueandthecoverage

properties

ofaCAarenot

affected(usually

arerepresentedwiththesymbol

*).

Definition

Oneelem

entof

aCAisredundant(wildcard)ifwecanchange

that

elem

entwithanyvalueof

itsalphabet

andthecoverage

properties

oftheCAarenot

affected.

16/72

For

instance

theCAACA(11;3,5,2)

donwloaded

from

theNIST

Repository

9has

onerowtotally

redundant.

TheCAB

indicates

with*theredundantelem

ents.

A=

00

01

00

01

01

01

00

00

11

10

01

01

11

10

11

11

10

11

00

01

11

01

01

01

11

10

10

0

B=

00

01

00

01

01

01

00

00

11

10

01

01

1∗

∗∗

∗∗

11

10

11

00

01

11

01

01

01

11

10

10

0

Figure

3:CAAcanbereducedin

onerowthrough

thedetection

of

redundantsymbolsshow

nin

CAB

as*

9http://math.nist.gov/coveringarrays/ipof/tables/table.3.2.htm

l17/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

18/72

CARepositories

CA

Repositories

Therearethreemainrepositoriesforuniform

CAs:

�NIST

Repository

10that

containsexplicitCAsconstructed

usingthe

IPOG-F

algorithm

for(v

={2,...,6})∧(t

={2,...,6}).

�CharlesColbourn

Repository

11that

lists

only

thebestknow

nsizesfor

CAs(v

={2,...,25})∧(t

={2,...,6},itdoestnotprovideexplicit

CAs.

�CinvestavCA

Repository

12currentlyitprovides

explicitCAs.

Itcontains

goodCAsfor(v

={2,3})∧(t

={2,...,6}).

10http://math.nist.gov/coveringarrays/

11http://www.public.asu.edu/∼ccolbou/src/tabby/catable.htm

l12http://www.tam

ps.cinvestav.m

x/∼jtj/authentication.php

19/72

20/72

21/72

22/72

23/72

24/72

25/72

26/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

27/72

ConstructionMethods

Dueto

thediffi

cultyofsolvingtheCACP

anumber

ofmethodshavebeen

developed.Themethodscanbeclassified

infive

maincategories:

�Exact

methods.

Thesemethodsguaranteeto

findan

optimalCA,butgiven

theexponential

search

spacethey

arepractical

only

forsm

allCAs.

�Greedymethods.

Thesemethodsdonotguaranteeto

findan

optimal

CA,

butthey

arevery

fast

andcanbeusedto

construct

anyCA.

�Metaheuristic

methods.

Thesemethodsdoes

notguaranteeto

findand

optimal

CA,they

take

moretimethan

thegreedymethodsandin

many

casesthey

givebetterCAsthan

theones

obtained

usinggreedymethods.

�Algebraicmethods.

Thealgebraicmethodsinvolved

form

ulasor

operations

withmathem

atical

objectssuch

asvectors,finitefields,

groupsor

another

(usually)sm

allcoveringarrays.

�Manipulationmethods.

Thesemethodsuse

coveringarrays

previouslycon-

structed

toconstruct

new

ones,or

transform

aCA

tomoresuitable

CA.

28/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

29/72

ManipulationofCAs

Therearesomeusefuloperationsthat

canbeappliedto

acoveringarrayprevi-

ouslyconstructed.Thissectiondescribes

fourofthem

:maxim

izationofconstant

rows,optimal

reduction,wildcard

detection,andfusion.

�Themaxim

izationofconstantrowsenable

that

theproduct

ofCAs,

and

thepow

eringofCAsproduce

redundantrowsthat

areelim

inated

easily.

�Theoptimal

reductionofCAs,

enable

toconstruct

smallCAstakingas

inputCAswithgreater

number

ofrowsandcolumns.

�Thewildcard

detectionofCAs,will

detectredundantelem

ents

inaCA.

�ThefusionofCAs,enableto

exploitsystem

aticallythewildcardsto

reduce

thesize

ofaCA

30/72

ManipulationofCAs

Maxim

izationofConstantRow

sin

aCA

�A

constantrow

inacoveringarrayisarow

havingthesamesymbolin

allits

elem

ents.Form

ally,thei-th

row

ofacoveringarrayA=

(ai,j)

ofdim

ensions

N×

kisconstantifai,j=

ai,0forj=

1,2

,...,k

−1

13.

�Bymeansofthethreeoperationsthatproduce

isomorphic

coveringarrays

itis

possible

toarrangethesymbolsofthecoveringarrayin

order

tomake

constant

someofitsrows.

�Theconstantrowsarevery

usefulforthemethodsofmultiplicationand

poweringofcoveringarrays,because

ifthecoveringarrays

usedhave

constant

rows,then

itispossible

todeletesomerowsin

theresultingcoveringarray.

�Thisproblem

canbesolved

inthedomain

ofgraphs.

Foreach

row

anodeis

created,andforeach

pairofrowsthataredistinct

(columnbycolumn)anedge

iscreated.Theproblem

isconverted

totheMAXCLIQ

UEproblem.

13Quiz-Ram

os[49]

31/72

0→

02

21

1→

21

10

2→

10

02

3→

10

20

4→

11

01

5→

12

12

6→

20

01

7→

21

22

8→

22

11

9→

00

12

10→

01

10

11→

02

00

38

10

652

01

11

79

4

00

00

11

11

22

22

22

01

21

20

20

12

12

20

11

02

10

10

02

12

01

11

00

21

32/72

ManipulationofCAs

ShorteningofaCA

�Given

acoveringarrayA

theOptimal

ShorteningofCoveringARrays

(OSCAR)problem

consistsin

findingasubmatrixB

ofadetermined

size

such

that

thenumber

ofmissingtuplesin

Bisminim

ized

14.

�Let

beδandΔ

twointegerssuch

that

0≤

δ≤

N−

vt,0≤

Δ≤

k−

t,withtheconditionthat

atleastoneofthem

isgreater

than

zero.

�TheOSCARproblem

consistsin

findingasubmatrixB

ofA

ofsize

(N−δ)

×(k

−Δ)such

that

thenumber

ofmissingtuplesin

Bisminim

al.

�Itwas

provedthat

theOSCARproblem

isNP-Complete

byreducingthe

MAXCOVERproblem

totheOSCARproblem.

�Thesearch

spaceoftheOSCARproblem

is�

NN−δ

��k

k−Δ

� .

14Carrizales-Turrubiates[50]

33/72

00

00

00

00

00

11

11

01

11

00

10

11

01

10

10

10

01

11

10

11

00

CA(6;2,7,2)

00

00

00

00

00

11

11

01

11

00

10

11

01

10

10

10

01

11

10

11

00

Row

andcolumnsto

delete

00

00

00

11

01

01

10

01

11

10

CA(5;2,4,2)

Figure

4:Thedeletionof

onerowandthreecolumnsof

thecovering

arrayCA(6;2,7,2)producesthecoveringarrayCA(5;2,4,2).

34/72

ManipulationofCAs

RedundantElements

Detectionin

aCA

Sometim

esacoveringarrayhasentriesthatcanbefreely

modified

withoutaffecting

thecoverageproperties

ofthecoveringarray,thatis,withoutaffectingthenumber

of

missingtuplesofthearray.

Theseentriesarecalled

wildcardsandarecommonly

representedbythesymbol∗.Figure

5showsattheleft

thecoveringarray

CA(7;2

,8,2),

andshowsatrightthesamecoveringarraywiththewildcardsit

contains.

00

10

10

00

01

00

10

11

11

11

00

10

00

10

11

11

11

01

11

01

10

00

01

00

00

11

00

01

0∗

1∗

10

00

01

00

∗0

1∗

11

11

00

10

00

10

11

11

11

01

11

01

10

00

01

00

00

∗1

00

∗1

Figure

5:Wildcardsin

thecoveringarrayCA(7;2,8,2).

35/72

Wildcard

detectionisvery

importantforsomepostoptimizationprocess

forcovering

arrays.A

postoptimizationprocess

isaprocess

thattriesto

reduce

thenumber

of

rowsofagiven

coveringarray.

Oneoftheseprocess

thatuseswildcardsisthemethod

ofNayeriet

al.

15

Amethodologyto

maximizethenumber

ofwildcardsin

acoveringarray

16proposed

threemain

steps:

�to

determinethetuplescoveredonly

once;

�to

determinetheunfixedsymbols;

�to

enumerate

allthepossible

wildcard

configurations.

Thefirstandsecondstepsareseen

inthenextfigure.Theelem

ents

thatpaticipate

in

t-wisecombinatioscoveredonce

areseen

inthesecondmatrix

as0or1,theunfixed

elem

ents

areshownin

thesecondmatrix

asU.Thethirdstep

canbeim

plemented

usingagreedyapproach

oranexact

approach,butin

anycase

both

approaches

tries

tominim

izetheU

elem

ents

thatbecomefixed(tosomevalueofZv),

andin

thisway

theremainingU

elem

ents

aremaximized

andconverted

towildcards(∗).

15Nayeriet

al.[51]

16Gonzalez-Hernandez

etal.[52]

36/72

00

10

10

00

01

00

10

11

11

11

00

10

00

10

11

11

11

01

11

01

10

00

01

00

00

11

00

01

0U

UU

1U

U0

01

00

U0

1U

11

11

00

10

00

1U

U1

1U

11

01

11

01

10

00

01

U0

00

U1

0U

U1

Figure

6:Theentriesof

thecoveringarrayCA(7;2,8,2)markedwithU

aretheentriesthat

may

becom

eawildcard.

37/72

ManipulationofCAs

FusionofaCA

Colbourn

17defined

thefusionofaCA

as:

CAN(t,k

,v)≤

CAN(t,k

,v+1)−

3ift=

2,k≤

v+

1,visaprimepower

2otherwise

(4)

Thebasicmechanism

ofthefusionoperatoristo

obtain

from

acoveringarrayA=

CA(N

;t,k

,v)another

coveringarrayB

=CA(M

;t,k

,v−

1)ofsm

aller

size

by

replacingtheoccurrencesofthesymbolvin

Aforsymbolsoftheset{0,1

,...,v

−2},

andbydeletingthreeortworowsaccordingthecasesofexpression(4).

17Colbourn

[53],andColbourn

etal.[54]

38/72

Ageneralizationofthefusionoperator

toCAs

18was

proposed.Thegeneraliza-

tionconsistsin

threemaincomponents:

�Awildcard

detector,that

maxim

izes

thewildcardsin

aCA.Theim

plemen-

tationofthiscomponentcould

begreedyor

exact.

�A

row

replacerthat

exploitstheredundantelem

ents

toreduce

therows

oftheCA.Thiscomponentcopiesnon-w

ildcard

elem

ents

tocorresponding

wildcard

elem

ents

(inthesamecolumn)in

order

toobtain

aredundant

row

that

canbeelim

inated.Thisprocedure

could

endwhen

aredundant

row

isfound,or

when

thebestnon-redundantrow

isfound(onerow

that

consumes

theless

number

ofwildcard

elem

ents).

Incase

aredundant

row

could

notbefound,aquasi-CA

with

theminim

um

missingt-wise

combinationsisdelivered.

�A

metaheuristic

algorithm

that

triesto

makezero

thenumber

ofmissing

tuples.

Thisprocedure

isrununtilthenumber

ofmissingtuplesiszero

oramaxim

um

timeisconsumed.

When

theorder

oftheinputCA

isgreater

than

theorder

oftheoutputCA

an

order

reductor

algorithm

isused.

18Rodriguez-Cristerna[55]

39/72

start

order

reductor

wildcard

detector

row

replacer

missing>

0?

simulatedannealing

missing>

0?

end

end

yes

no

yes

yes

no

40/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

41/72

PostprocessigNISTRepository

�AspartoftheAutomated

CombinatorialTestingforSoftware(A

CTS)

project

atNIST,arepository

ofcoveringarrays

ispublicly

available.An

importantopportunityistheexploitationoftheredundancy

ofcoverage

that

thecoveringarrays

havein

order

toreduce

itssize.

�TheapplicationoftheGeneralized

fusionoperator

producesalotof

improvements

oftheCA

oftheNIST

Repository.

�TheNIST

repository

was

processed

usingthegeneralized

fusionandWe

havefound349new

upper

bounds.

�Wehaveem

ployedmorethan

1.5

millionofhours

oftheXiuhcoatlcluster

oftheCINVESTAV.

�A

totalof21,964CAswereprocessed

and20,956wereim

proved.

�Anaveragereductionof3.85%

was

attained.

42/72

0 0.5

1 1.5

2 2.5

3 3.5 2

2.5

3

3.5

4

4.5

5

5.5

6

0

5

10

15

20

25

−Δ

t=2

v2

v3

v4

v5

v6

log(k)

v

−Δ

43/72

0.5

1

1.5

2

2.5

3

3.5 2

2.5

3

3.5

4

4.5

5

5.5

6

0

5

10

15

20

25

30

−Δ

t=3

v2

v3

v4

v5

v6

log(k)

v

−Δ

44/72

0.5

1

1.5

2

2.5

3 2

2.5

3

3.5

4

4.5

5

5.5

6

0

5

10

15

20

25

30

−Δ

t=4

v2

v3

v4

v5

v6

log(k)

v

−Δ

45/72

0.6 0.8 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2

2.5

3

3.5

4

4.5

5

5.5

6

0

5

10

15

20

25

−Δ

t=5

v2

v3

v4

v5

v6

log(k)

v

−Δ

46/72

0.8

1

1.2

1.4

1.6

1.8

2 2

2.5

3

3.5

4

4.5

5

0

2

4

6

8

10

12

14

16

18

20

−Δ

t=6

v2

v3

v4

v5

log(k)

v

−Δ

47/72

v3t4

v3t4

v6t4

Id.

kIPOG-F

FG-F1

Id.

kIPOG-F

FG-F1

Id.

kIPOG-F

FG-F1

1315

968

964

44

377

1011

1003

85

85

11441

11384

2316

969

964

45

379

1013

1007

86

86

11484

11407

3317

969

963

46

386

1017

1009

87

87

11533

11478

4318

971

965

47

405

1027

1024

88

88

11577

11504

5319

971

963

48

439

1046

1045

89

89

11625

11581

6320

971

963

49

447

1052

1050

90

90

11666

11591

7321

974

965

v6t4

91

91

11710

11630

8322

974

966

Id.

kIPOG-F

FG-F1

92

92

11753

11671

9323

975

970

50

49

9323

9212

93

93

11790

11729

10

324

976

965

51

50

9393

9294

94

94

11833

11762

11

325

976

970

52

51

9466

9397

95

95

11874

11800

12

326

976

974

53

52

9550

9463

96

96

11913

11884

13

327

977

973

54

53

9623

9540

97

97

11956

11918

14

328

978

969

55

55

9762

9673

98

98

11997

11924

15

329

979

975

56

56

9828

9742

99

99

12038

11949

16

330

979

976

57

57

9900

9813

100

100

12085

12003

17

331

981

977

58

58

9964

9869

101

101

12120

12036

18

332

981

973

59

59

10032

9948

102

102

12148

12140

19

333

983

975

60

60

10097

10013

103

103

12194

12120

20

335

983

979

61

61

10163

10067

104

104

12231

12185

21

336

983

978

62

62

10219

10142

105

105

12267

12196

22

337

984

977

63

63

10282

10198

106

106

12306

12240

23

338

985

977

64

64

10347

10250

107

107

12343

12268

24

339

986

977

65

65

10398

10328

108

109

12408

12388

25

340

987

978

66

66

10463

10368

109

110

12445

12380

26

341

987

979

67

67

10520

10441

110

112

12517

12449

27

342

987

981

68

68

10578

10478

111

116

12651

12593

28

343

990

985

69

69

10633

10572

112

118

12716

12698

29

345

991

984

70

70

10693

10599

113

120

12785

12734

30

346

992

985

71

71

10745

10676

114

121

12816

12757

31

347

992

985

72

72

10798

10744

115

128

13036

13007

32

348

992

987

73

73

10850

10758

116

129

13062

13056

33

349

992

986

74

74

10909

10821

117

133

13192

13146

34

351

993

991

75

75

10958

10882

118

149

13634

13606

35

353

994

987

76

76

11012

10959

v3t5

36

360

999

994

77

77

11057

10992

Id.

kIPOG-F

FG-F1

37

361

1001

995

78

78

11110

11017

119

35

1867

1826

38

362

1001

996

79

79

11158

11100

120

36

1895

1850

39

364

1002

997

80

80

11203

11121

121

37

1920

1882

40

368

1007

1002

81

81

11253

11187

122

38

1947

1909

41

370

1008

1001

82

82

11303

11219

123

39

1974

1933

42

373

1009

1003

83

83

11353

11282

124

40

1997

1949

43

375

1009

1005

84

84

11397

11319

125

41

2023

1975

47/72

v3t5

v3t5

v4t5

Id.

kIPOG-F

FG-F1

Id.

kIPOG-F

FG-F1

Id.

kIPOG-F

FG-F1

126

42

2046

2002

169

85

2739

2700

210

45

9227

9104

127

43

2070

2022

170

86

2749

2706

211

46

9320

9176

128

44

2091

2050

171

87

2762

2728

212

47

9406

9262

129

45

2112

2071

172

88

2770

2736

213

48

9501

9357

130

46

2130

2086

173

89

2783

2747

214

49

9588

9453

131

47

2150

2112

174

90

2792

2761

215

50

9673

9520

132

48

2174

2134

175

91

2805

2772

216

51

9755

9621

133

49

2191

2154

176

92

2815

2783

217

52

9835

9682

134

50

2213

2182

177

93

2825

2798

218

53

9922

9769

135

51

2231

2198

178

94

2836

2797

219

54

9998

9849

136

52

2251

2217

179

95

2847

2813

220

55

10079

9927

137

53

2269

2232

180

96

2857

2819

221

56

10155

10007

138

54

2289

2246

181

97

2868

2834

222

57

10232

10082

139

55

2309

2277

182

98

2877

2838

223

59

10379

10231

140

56

2327

2281

183

99

2885

2854

224

60

10454

10302

141

57

2342

2304

184

100

2895

2862

225

62

10590

10442

142

58

2358

2316

185

101

2909

2871

226

63

10650

10509

143

59

2373

2335

186

102

2920

2878

v5t5

144

60

2394

2347

187

103

2928

2898

Id.

kIPOG-F

FG-F1

145

61

2408

2370

188

104

2938

2901

227

21

18779

18260

146

62

2425

2387

189

105

2945

2908

228

22

114775

111818

147

63

2440

2397

190

107

2962

2928

229

23

118587

115802

148

64

2451

2413

v4t5

230

24

122201

119500

149

65

2468

2439

Id.

kIPOG-F

FG-F1

231

25

125683

123108

150

66

2482

2447

191

26

6957

6775

v6t5

151

67

2498

2459

192

27

7116

6937

Id.

kIPOG-F

FG-F1

152

68

2512

2477

193

28

7267

7088

232

17

40334

38976

153

69

2527

2487

194

29

7414

7247

233

18

42102

40820

154

70

2542

2504

195

30

7555

7379

234

19

43833

42554

155

71

2555

2518

196

31

7691

7527

235

20

45425

44224

156

72

2573

2531

197

32

7816

7649

236

21

46970

45784

157

73

2584

2546

198

33

7939

7782

237

22

48479

47352

158

74

2597

2564

199

34

8064

7907

238

23

49924

48838

159

75

2609

2588

200

35

8183

8023

239

24

51287

50180

160

76

2625

2593

201

36

8301

8137

240

25

52604

51505

161

77

2639

2608

202

37

8420

8259

241

26

53850

52814

162

78

2648

2615

203

38

8530

8376

242

27

55069

54032

163

79

2661

2625

204

39

8629

8477

243

28

56225

55275

164

80

2673

2635

205

40

8737

8583

244

29

57363

56353

165

81

2686

2651

206

41

8847

8693

245

30

58468

57503

166

82

2700

2668

207

42

8945

8791

246

31

59529

58576

167

83

2710

2677

208

43

9035

8890

247

32

60570

59612

168

84

2725

2684

209

44

9137

8979

248

33

61562

60608

47/72

v6t5

v2t6

v4t6

Id.

kIPOG-F

FG-F1

Id.

kIPOG-F

FG-F1

Id.

kIPOG-F

FG-F1

249

34

62527

61612

290

79

782

770

329

26

33369

32513

250

35

63471

62557

291

80

785

771

330

27

34187

33356

251

36

64399

63519

292

81

791

772

331

28

35006

34198

v2t6

293

82

794

778

332

29

35791

34971

Id.

kIPOG-F

FG-F1

294

83

796

787

333

30

36570

35778

252

41

572

556

295

84

800

784

334

31

37305

36515

253

42

579

561

296

85

804

790

335

32

38015

37255

254

43

590

570

297

86

809

799

v5t6

255

44

594

580

v3t6

Id.

kIPOG-F

FG-F1

256

45

603

589

Id.

kIPOG-F

FG-F1

336

11

56615

52471

257

46

611

596

298

26

5709

5544

337

12

63620

59622

258

47

617

602

299

27

5853

5667

338

13

70190

66275

259

48

625

612

300

28

6003

5827

339

14

76390

72680

260

49

630

617

301

29

6150

5969

340

15

82139

78480

261

50

636

623

302

30

6281

6103

341

16

87559

84102

262

51

643

629

303

31

6413

6245

342

18

97605

94263

263

52

650

636

304

32

6535

6348

343

19

102208

98994

264

53

656

641

305

33

6656

6461

344

20

106642

103514

265

54

662

644

306

34

6772

6583

345

21

110842

107773

266

55

667

653

307

35

6877

6715

346

22

114775

111818

267

56

672

655

308

36

6989

6832

347

23

118587

115802

268

57

677

663

309

37

7092

6932

348

24

122201

119500

269

58

683

667

310

38

7194

7036

349

25

125683

123108

270

59

689

671

311

39

7293

7131

271

60

695

680

312

40

7391

7233

272

61

699

687

313

41

7490

7315

273

62

703

686

314

42

7574

7411

274

63

709

697

315

43

7672

7506

275

64

715

699

316

44

7757

7600

276

65

721

704

317

45

7845

7702

277

66

725

709

318

46

7938

7766

278

67

728

710

319

47

8013

7856

279

68

732

718

320

50

8256

8108

280

69

738

724

321

51

8333

8179

281

70

743

729

v4t6

282

71

747

734

Id.

kIPOG-F

FG-F1

283

72

751

736

322

19

26392

25430

284

73

755

743

323

20

27534

26564

285

74

761

749

324

21

28625

27676

286

75

766

751

325

22

29640

28735

287

76

770

755

326

23

30636

29720

288

77

773

758

327

24

31591

30724

289

78

777

760

328

25

32501

31654

47/72

TableofContents

Introduction

BasicDefinitions

CARepositories

ConstructionMethods

Manipulation

ofCAs

PostprocessingNISTRepository

Conclusions

48/72

Conlusions

�TheCoveringArrays(C

A)representanexcelentoptionwhen

itisdesired

toguaranteeacertain

levelofcoveragein

thetestingofasoftwarecomponentand

touse

asm

allnumber

oftest

cases.

�Wehave

presentedbasicdefinitionsneeded

tounderstandmuch

oftheresearch

relatedto

coveringarrays.

�Wehave

grouped

alotofresearch

reportsrelatedto

theconstructionofCAs,

thegroupswere:

Exact

methods,GreedyMethods,Metaheuristic

Methods,and

ManipulationMethods.

�Thegeneralizedfusionoperatorisbasedontheexploitationofthewildcardsof

aCA

inorder

toreduce

thenumber

ofrowsoftheCA.

�Wehave

processed

alltheNIST

CA

Repository

usingmore

than1.5

millonof

CPU

hours

(XiuhcoatlCinvestavCluster)andhave

found349new

upper

bounds.

�Itisim

portantto

highlightthattheexploitationoftheredundancy

ofcoverage

inCAscanenable

thecreationofbetterCAs(w

ithlower

number

ofrows).

49/72

Thanks!!!

50/72

ReferencesI

[1]

B.HailpernandP.Santhanam

,“Softwaredebugging,

testing,

andverification,”

IBM

SystemsJournal,vol.41,no.

1,pp.4

–12,

2002.

[2]

A.Hartm

an,“Softwareandhardwaretestingusing

combinatorialcoveringsuites,”

inGraphTheory,

CombinatoricsandAlgorithms,ser.Operations

Research/C

omputerScience

Interfaces

Series.

SpringerUS,

2005,vol.34,pp.237–266.

[3]

D.Cohen,S.Dalal,J.

Parelius,andG.Patton,“T

he

combinatorialdesignapproach

toautomatictest

generation,”

Software,

IEEE,vol.13,no.

5,pp.83

–88,

sep1996.

[4]

D.Cohen,S.Dalal,M.Fredman,andG.Patton,“T

heaetg

system

:Anapproach

totestingbased

oncombinatorial

design,”

IEEETransactionson

SoftwareEngineering,

vol.23,

no.

7,pp.437–444,

jul1997.

51/72

ReferencesII

[5]

K.Bush,“O

rthogonalarrays

ofindex

unity,”TheAnnalsof

Mathem

atical

Statistics,vol.23,no.

3,pp.426–434,1952.

[6]

J.Torres-Jimenez,N.Rangel-Valdez,L.Gonzalez-Hernandez,

andH.Avila-George,“C

onstructionof

logarithm

tablesfor

GaloisFields,”International

Journal

ofMathem

atical

Education

inScience

andTechnology,vol.42,no.

1,pp.

91–102,Oct.2010.

[7]

CryptographyandSecurity

inCom

puting.

InTech,2012,ch.

Constructionof

OrthogonalArraysof

Index

UnityUsing

Logarithm

TablesforGaloisFields.

[8]

C.Colbourn,“C

ombinatorialaspects

ofcoveringarrays,”

Le

Matem

atiche,vol.59,no.

1,2,

pp.125–172,

2004.

[9]

Foundationsof

Probability.

Holden-Day,1970.

52/72

ReferencesIII

[10]G.Katona,

“Twoapplications(for

search

theory

andtruth

functions)

ofsperner

typetheorems,”PeriodicaMathem

atica

Hungarica,1973.

[11]

D.Kleitman

andJ.

Spencer,“Fam

ilies

ofk-independent

sets,”

DiscreteMathem

atics,1973.

[12]

D.Kuhn,J.

Higdon,J.

Law

rence,R.Kacker,andL.Y.,

“Com

binatorialmethodsforeventsequence

testing,”in

Workshop

onCom

binatorialTesting(CT)In

conjunctionwith

International

Conference

onSoftwareTesting.

[13]J.

Yan

andJ.

Zhang,

“Backtrackingalgorithmsandsearch

heuristicsto

generatetest

suites

forcombinatorialtesting,”in

Proceedings

ofthe30th

Annual

International

Com

puter

SoftwareandApplicationsConference

-Volume01,ser.

COMPSAC’06.

Washington,DC,USA:IEEECom

puter

53/72

ReferencesIV

Society,2006,pp.385–394.

[Online].Available:

http://dx.doi.org/10.1109/C

OMPSAC.2006.33

[14]J.

Bracho-Rios,J.Torres-Jimenez,andE.Rodriguez-Tello,“A

new

backtrackingalgorithm

forconstructingbinarycovering

arrays

ofvariablestrength,”

inMICAI2009:Advancesin

Artificial

Intelligence,ser.Lecture

Notes

inCom

puterScience,

A.Aguirre,R.Borja,andC.Garcia,

Eds.

SpringerBerlin

/Heidelberg,

2009,vol.5845,pp.397–407.

[Online].Available:

http://dx.doi.org/10.1007/978-3-642-05258-3

35

[15]B.Hnich,S.D.Prestwich,E.Selensky,andB.M.Smith,

“Constraintmodelsforthecoveringtest

problem,”

Constraints,vol.11,no.

2-3,

pp.199–219,

Jul.2006.[Online].

Available:http://dx.doi.org/10.1007/s10601-006-7094-9

54/72

ReferencesV

[16]D.Lopez-Escogido,

J.Torres-Jimenez,E.Rodriguez-Tello,

andN.Rangel-Valdez,“Strengthtwocoveringarrays

constructionusingasatrepresentation,”

inMICAI,2008,pp.

44–53.

[17]M.Banbara,

H.Matsunaka,

N.Tam

ura,andK.Inoue,

“Generatingcombinatorialtest

casesby

efficientsat

encodings

suitableforcdclsatsolvers,”in

Proceedings

ofthe

17th

international

conference

onLogicforprogramming,

artificial

intelligence,andreasoning,

ser.LPAR’10.

Berlin,

Heidelberg:

Springer-Verlag,

2010,

pp.112–126.

[18]K.Meagher,“N

on-isomorphicgenerationof

coveringarrays,”

University

ofRegina,

Tech.Rep.,2002.

[19]R.Bryce

andC.Colbourn,“T

hedensity

algorithm

for

pairwiseinteractiontesting,”SoftwareTesting,

Verification

andReliability,vol.17,no.

3,pp.159–182,

2007.

55/72

ReferencesVI

[20]Y.Lei,R.Kacker,D.Kuhn,V.Okun,andJ.Law

rence,“Ipog:

ageneralstrategy

fort-way

softwaretesting,”in

Proceedings

ofthe14th

Annual

IEEEInternational

Conference

and

Workshopson

theE

ngineeringof

Com

puter-Based

Systems.

ECBS’07.,Tucson,AZ,USA,mar

2007,pp.549–556.

[21]

M.Forbes,J.

Law

rence,Y.Lei,R.Kacker,andD.Kuhn,

“Refiningthein-param

eter-order

strategy

forconstructing

coveringarrays,”

Journal

ofResearchof

theNational

Institute

ofStandardsandTechnology,vol.113,

no.

5,pp.287–297,

2008.

56/72

ReferencesVII

[22]Y.Lei,R.Kacker,D.R.Kuhn,V.Okun,andJ.

Law

rence,

“Ipog:Ageneralstrategy

fort-way

softwaretesting,”in

Proceedings

ofthe14th

Annual

IEEEInternational

Conference

andWorkshopson

theEngineeringof

Com

puter-Based

Systems,ser.ECBS’07.

Washington,DC,

USA:IEEECom

puterSociety,2007,pp.549–556.

[Online].

Available:http://dx.doi.org/10.1109/E

CBS.2007.47

[23]

A.CalvagnaandA.Gargantini,“T

-wisecombinatorial

interactiontest

suites

constructionbased

oncoverage

inheritance,”

SoftwareTesting,

Verification

andReliability,

vol.22,no.

7,pp.507–526,

2012.[Online].Available:

http://dx.doi.org/10.1002/stvr.466

57/72

ReferencesVIII

[24]E.Covarrubias-Flores,“C

alculodecoveringarrays

binariosde

fuerza

variable,usandounalgoritm

oderecocidosimulado,”

Master’sthesis,CentrodeInvestigacionydeEstudios

Avanzados

delInstituto

PolitecnicoNacional,2008.

[25]J.

Torres-Jimenez

andE.Rodriguez-Tello,“N

ewboundsfor

binarycoveringarrays

usingsimulatedannealing,”Inform

ation

Sciences,vol.185,

no.

1,pp.137–152,

February2012.

[26]

H.Avila-George,J.

Torres-Jimenez,andV.Hernandez,

“Parallelsimulatedannealingforthecoveringarrays

constructionproblem,”

inToappearin

the18th

International

Conference

onParallelandDistributedProcessingTechniques

andApplications,2012.

58/72

ReferencesIX

[27]H.Avila-George,J.

Torres-Jimenez,andV.Hernandez,“N

ewboundsforternarycoveringarrays

usingaparallelsimulated

annealing,”Mathem

atical

Problemsin

Engineering,

Accepted

onJuly07,2012.

[28]K.Nurm

ela,

“Upper

boundsforcoveringarrays

bytabu

search,”

DiscreteAppliedMathem

atics,vol.138,

no.

1-2,

pp.

143–152,

2004,optimalDiscreteStructuresandAlgorithms.

[29]

L.Gonzalez-Hernandez,N.Rangel-Valdez,and

J.Torres-Jimenez,“C

onstructionof

mixed

coveringarrays

ofvariablestrengthusingatabusearch

approach,”

inCombinatorialOptimizationandApplications,ser.Lecture

Notes

inCom

puterScience,W.WuandO.Daescu,Eds.

SpringerBerlin

/Heidelberg,

2010,vol.6508,pp.51–64.

[Online].Available:

http://dx.doi.org/10.1007/978-3-642-17458-2

6

59/72

ReferencesX

[30]J.

Stardom

,“M

etaheuristicsandthesearch

forcoveringand

packingarrays,”

Master’sthesis,Simon

FraserUniversity,

2001.

[31]T.Shiba,

T.Tsuchiya,

andT.Kikuno,

“Usingartificiallife

techniques

togeneratetest

casesforcombinatorialtesting,”

inCom

puterSoftwareandApplicationsConference,2004.

COMPSAC2004.Proceedings

ofthe28th

Annual

International,vol.1,

sept.2004,

pp.72–77.

[32]

D.T.TangandL.S.Woo,

“Exhaustivetest

pattern

generationwithconstantweightvectors,”IEEETrans.

Comput.,vol.32,no.

12,pp.1145–1150,

Dec.1983.[Online].

Available:http://dx.doi.org/10.1109/T

C.1983.1676175

60/72

ReferencesXI

[33]J.

Martinez-PenaandJ.

Torres-Jimenez,“A

branch

and

boundalgorithm

forternarycoveringarrays

constructionusing

trinom

ialcoeffi

cients,”

Researchin

Com

putingScience,

vol.49,pp.61–71,

2010.

[34]J.

Martinez-Pena,

J.Torres-Jimenez,N.Rangel-Valdez,and

H.Avila-George,“A

heuristicapproach

forconstructing

ternarycoveringarrays

usingtrinom

ialcoeffi

cients,”

inProceedings

ofthe12th

Ibero-American

conference

onAdvancesin

artificial

intelligence,ser.IBERAMIA’10.

Berlin,

Heidelberg:

Springer-Verlag,

2010,

pp.572–581.

[Online].

Available:

http://dl.acm.org/citation.cfm

?id=1948131.1948203

[35]C.Colbourn,“C

overingarrays

from

cyclotom

y,”Designs,

Codes

andCryptography,vol.55,pp.201–219,

2010.[Online].

Available:http://dx.doi.org/10.1007/s10623-009-9333-8

61/72

ReferencesXII

[36]K.Meagher

andB.Stevens,“G

roupconstructionof

covering

arrays,”

Journal

ofCom

binatorialDesigns,vol.13,no.

1,pp.

70–77,2005.[Online].Available:

http://dx.doi.org/10.1002/jcd.20035

[37]

J.Lobb,C.Colbourn,P.Danziger,B.Stevens,and

J.Torres-Jimenez,“C

over

startersforcoveringarrays

ofstrengthtwo,”DiscreteMathem

atics,2011,in

press.

[38]

N.J.

A.Sloane,“C

overingarrays

andintersectingcodes,”

Journal

ofCom

binatorialDesigns,vol.1,

no.

1,pp.51–63,

1993.

[39]

G.Roux,

“k-proprietesdansdes

tableauxdencolonnes;cas

particulierdelak-surjectivite

etdelak-permutivite,”

Ph.D.

dissertation,University

ofParis,1987.

62/72

ReferencesXIII

[40]M.ChateauneufandD.Kreher,“O

nthestateof

strength-threecoveringarrays,”

Journal

ofCombinatorial

Designs,vol.10,no.4,

pp.217–238,2002.

[41]M.B.Cohen,C.J.

Colbourn,andA.C.Lin,“C

onstructing

strengththreecoveringarrays

withaugm

entedannealing,”

DiscreteMathem

atics,vol.308,

no.

13,pp.2709–2722,2008.

[42]

C.J.

Colbourn,S.S.Martirosyan,T.Trung,

andR.A.

Walker,II,“R

oux-typeconstructionsforcoveringarrays

ofstrengthsthreeandfour,”Des.Codes

Cryptography,vol.41,

no.1,

pp.33–57,

Oct.2006.[Online].Available:

http://dx.doi.org/10.1007/s10623-006-0020-8

[43]S.Martirosyan

andT.v.

Trung,

“Ont-coveringarrays,”

Designs,Codes

andCryptography,vol.32,pp.323–339,2004,

10.1023/B:DESI.0000029232.40302.6d.[Online].Available:

http://dx.doi.org/10.1023/B

:DESI.0000029232.40302.6d

63/72

ReferencesXIV

[44]C.J.

Colbourn,S.S.Martirosyan,G.L.Mullen,D.Shasha,

G.B.Sherwood,andJ.

L.Yucas,“P

roductsof

mixed

coveringarrays

ofstrengthtwo,”Journal

ofCom

binatorial

Designs,vol.14,no.2,

pp.124–138,2006.

[45]

A.Hartm

an,“Softwareandhardwaretestingusing

combinatorialcoveringsuites,”

inGraphTheory,

Com

binatoricsandAlgorithms,ser.Operations

Research/C

omputerScience

Interfaces

Series,M.C.Golumbic

andI.B.-A.Hartm

an,Eds.

SpringerUS,2005,vol.34,pp.

237–266.[Online].Available:

http://dx.doi.org/10.1007/0-387-25036-0

10

[46]

H.Avila-George,“V

erificacion

decoveringarrays

utilizando

supercomputacion

ycomputacion

grid,”

Master’sthesis,

Universidad

PolitecnicadeValencia,

2010.

64/72

ReferencesXV

[47]H.Avila-George,J.

Torres-Jimenez,V.Hernandez,and

N.Rangel-Valdez,“A

parallelalgorithm

fortheverification

ofcoveringarrays,”

inProceedings

ofTheInternational

Conference

onParallelandDistributedTechniques

and

Applications(PDPTA2011),Las

Vegas

Nevada,

USA,18-21

July2011,pp.879–885.

[Online].Available:

http://w

orld-com

p.org/p2011/P

DP8061.pdf

[48]——,“V

erification

ofgeneralandcycliccoveringarrays

using

grid

computing,”in

Proceedings

oftheThirdinternational

conference

onDatamanagem

entin

grid

andpeer-to-peer

system

s,ser.Globe’10.

Berlin,Heidelberg:

Springer-Verlag,

2010,pp.112–123.

[Online].Available:

http://dl.acm.org/citation.cfm

?id=1885229.1885242

65/72