23
The irace package: tutorial Leslie Pérez Cáceres IRIDIA, Université libre de Bruxelles Matheuris*cs 2016
The irace package: tutorial
Leslie Pérez Cáceres IRIDIA, Université libre de Bruxelles
Matheuris*cs 2016
Overview
• ConfiguraCon scenario • Irace opCons • ExecuCng irace • Results
2 irace tutorial – MatheurisCcs 2016
The irace package
• Implemented in R – Portable – No need to know R
• We will use irace 2.0 – Small changes from previous version
3 irace tutorial – MatheurisCcs 2016
irace
• What is needed to use it?
4
Instance set
Parameters definiCon
ConfiguraCon scenario
Target Runner
irace
Instance Parameter seWng Performance measure
irace tutorial – MatheurisCcs 2016
ConfiguraCon scenario
5
• Target algorithm – Parameter search space definiCon – Target runner
• Instances • ConfiguraCon opCons
irace tutorial – MatheurisCcs 2016
Scenario: target algorithm parameters
• Parameter space definiCon – Select the parameters of interest
• You can choose to expose “constants” – Choose the most adequate type for each parameter
– Choose a set or a range of values – Describe the dependencies between parameters
6 irace tutorial – MatheurisCcs 2016
• Available parameter types: – Categorical: Set of values that have no relaConship between them.
– Ordinal: Set of values that have a relaConship between them
– Numerical: Define a range of possible values • Real • Integer
Scenario: target algorithm parameters
7 irace tutorial – MatheurisCcs 2016
Example ACOTSP framework
• parameter: algorithm • descripCon: selects an ACO algorithm to apply • values: {as, eas, ras, mmas, acs} • dependencies: none
Categorical
Example CPLEX
• parameter: MIP subproblem algorithm • descripCon: conCnuous opCmizer to be used to
solve the subproblems in a MIP, aber the iniCal relaxaCon
• values: {Primal simplex, Dual simplex, Network simplex, Barrier, SiKing}
• dependencies: none
Categorical
Example CPLEX • parameter: MIP implied bound cuts switch • descripCon: whether or not to generate implied
bound cuts for the problem • values: {none, moderately, aggressively} • dependencies: none
Ordinal
Example ACOTSP framework
• parameter: alpha • descripCon: value for the alpha parameter • values: (0.00, 5.00) • dependencies: Used for all algorithms none
Numerical: Real
Example ACOTSP framework
• parameter: ras ranks • descripCon: ranks to be use to update the
pheromone • values: (0, 100) • dependencies: Only valid for Rank-‐based AS
algorithm %in% c(“ras”)
Numerical: Integer
• Available parameter types: – Categorical, Ordinal, Numerical
Scenario: target algorithm parameters
8 irace tutorial – MatheurisCcs 2016
Scenario: target algorithm parameters
• Dependencies ACOTSP
9
algorithm localsearch alpha beta rho ants
nnls dlb q0 rasranks eliCstants
irace tutorial – MatheurisCcs 2016
Scenario: target algorithm parameters # PARAMETER FILE FOR THE ACOTSP SOFTWARE # name switch type values [conditions (using R syntax)] algorithm "-‐-‐" c (as,mmas,eas,ras,acs) localsearch "-‐-‐localsearch " c (0, 1, 2, 3) alpha "-‐-‐alpha " r (0.00, 5.00) beta "-‐-‐beta " r (0.00, 10.00) rho "-‐-‐rho " r (0.01, 1.00) ants "-‐-‐ants " i (5, 100) nnls "-‐-‐nnls " i (5, 50) | localsearch %in% c(1, 2, 3) q0 "-‐-‐q0 " r (0.0, 1.0) | algorithm %in% c("acs") dlb "-‐-‐dlb " c (0, 1) | localsearch %in% c(1,2,3) rasrank "-‐-‐rasranks " i (1, 100) | algorithm %in% c("ras") elitistants "-‐-‐elitistants ” i (1, 750) | algorithm %in% c("eas")
10 irace tutorial – MatheurisCcs 2016
Scenario: target algorithm runner
• Script, program or R funcCon • Interface between irace and the target algorithm
• Input: – Instance – Seed – Set of parameters in the [flag value] format
• Output: EvaluaCon [Time]
11 irace tutorial – MatheurisCcs 2016
Scenario: target algorithm runner
• Executable program/script input: targetRunner <configuraCon id> <instance id> <seed> <instance> [<extra instance params>] <command line>
12
targetRunner 1 4 1465 ~/instances/instance1.txt -‐as -‐-‐localsearch 1 -‐-‐alpha 0.3 –beta 0.4 …
irace tutorial – MatheurisCcs 2016
Scenario: target algorithm runner #!/bin/bash # Input CANDIDATEID=$1 INSTANCEID=$2 SEED=$3 INSTANCE=$4 shift 4 CAND PARAMS=$* STDOUT="c${CANDIDATEID}-‐${INSTANCEID}.stdout" FIXED PARAMS=" -‐-‐time 1 -‐-‐tries 1 -‐-‐quiet " # Execute the algorithm ./acotsp $FIXED PARAMS -‐i $INSTANCE $CAND PARAMS 1> $STDOUT # Obtain the evaluation of the execution COST=$(grep -‐oE ’Best [-‐+0-‐9.e]+’ $STDOUT |cut –d ’ ’ -‐f2) if ! [[ "${COST}" = ̃ ˆ[-‐+0-‐9.e]+$ ]] ; then error "${STDOUT}: Output is not a number" fi echo "${COST}" exit 0
13 irace tutorial – MatheurisCcs 2016
Scenario: instances
• Choose a representaCve set of problem instances
• Instances can be – Files (directory or a file list) – FuncCons or parameters for an instance generator
14 irace tutorial – MatheurisCcs 2016
Scenario: irace opCons
• Instance sampling • StochasCc / determinisCc algorithms • StaCsCcal test: F-‐test or t-‐test • Parallel execuCon • Retrial of failed execuCons • TesCng of final configuraCons • Recovery of configuraCon process
15 irace tutorial – MatheurisCcs 2016
Exercise: ACOTSP
• Follow the instrucCons provided in
hop://iridia.ulb.ac.be/~lperez/matheurisCcs2016-‐irace/
• You can configure your own algorithm! – Install R – Install irace – Prepare scenario – Execute irace
16 irace tutorial – MatheurisCcs 2016
Exercise: ACOTSP # 2016-‐09-‐04 01:00:29 CEST: Iteration 1 of 5 # experimentsUsedSoFar: 0 # remainingBudget: 300 # currentBudget: 60 # nbConfigurations: 10 Markers: x No test is performed. -‐ The test is performed and some configurations are discarded. = The test is performed but no configuration is discarded. ! The test is performed and configurations could be discarded but elite configurations are preserved. +-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐+-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐+ | | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar| +-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐+-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐+ |x| 1| 10| 4| 29041929.00| 10|00:00:12| NA| NA| NA| |x| 2| 10| 4| 33245942.50| 20|00:00:16|+0.92|0.96|0.0041| |x| 3| 10| 4| 33240741.00| 30|00:00:14|+0.94|0.96|0.0035| |x| 4| 10| 4| 33175378.50| 40|00:00:14|+0.95|0.97|0.0034| |-‐| 5| 1| 4| 34060122.40| 50|00:00:16| NA| NA| NA| +-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐-‐-‐+-‐-‐-‐-‐-‐+-‐-‐-‐-‐+-‐-‐-‐-‐-‐-‐+ Best configuration: 4 mean value: 34060122.40 Description of the best configuration: .ID. algorithm localsearch alpha beta rho ants nnls dlb q0 rasrank elitistants .PARENT. 4 4 ras 3 1.94 5.97 0.46 78 12 1 NA 22 NA NA
Irace: Analysis and recent developments – COMEX meeCng 17
Exercise: ACOTSP
• The results are saved in a Rdata file. • All the test performed by irace are saved. – Can be used to analyse the parameter search space.
• Frequency plots of parameter values sampled
Irace: Analysis and recent developments – COMEX meeCng 18
> library("irace") > load("exec-‐dir/irace.Rdata") > parameterFrequency(iraceResults$allConfigurations, + iraceResults$parameters)
Exercise: ACOTSP
19
as eas ras acs
algorithm
values
Freq
uenc
y
040
100
0 1 2 3
localsearch
values
Freq
uenc
y
040
100
alpha
values
Prob
abilit
y de
nsity
0 1 2 3 4 5
0.00
0.20
beta
values
Prob
abilit
y de
nsity
0 2 4 6 8 10
0.00
0.10
rho
valuesPr
obab
ility
dens
ity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
1.5
ants
values
Prob
abilit
y de
nsity
20 40 60 80 100
0.00
00.
020
nnls
values
Prob
abilit
y de
nsity
10 20 30 40 50
0.00
0.03
q0
values
Prob
abilit
y de
nsity
0.0 0.2 0.4 0.6 0.8 1.0
0.0
1.5
0 1 <NA>
dlb
values
Freq
uenc
y
060
rasrank
values
Prob
abilit
y de
nsity
0 20 40 60 80 100
0.00
00.
015
elitistants
values
Prob
abilit
y de
nsity
0 200 400 600
0.00
000.
0020
irace tutorial – MatheurisCcs 2016
Exercise: ACOTSP
• Visualising configuraCons
20
> library("irace") > load("exec-‐dir/irace.Rdata") > last <-‐ length(iraceResults$iterationElites) > conf <-‐ getConfigurationByIteration(iraceResults = iraceResults, + iterations = c(last -‐ 1, last)) > parallelCoordinatesPlot (conf, iraceResults$parameters, param_names = c("algorithm", "alpha", "beta", "rho", "q0"), hierarchy = FALSE)
irace tutorial – MatheurisCcs 2016
Exercise: ACOTSP
21
Parameters parallel coordinates
algorithm
alpha
beta rho q0
as
mmas
eas
ras
acs
NA
0
1
2
3
4
5
<NA>
0
2
4
6
8
10
<NA>
0
0.2
0.4
0.6
0.8
1
<NA>
0
0.2
0.4
0.6
0.8
1
<NA>
irace tutorial – MatheurisCcs 2016
Exercise: ACOTSP
• TesCng of the best final configuraCons or best iteraCon configuraCons can be performed aber the execuCon.
• TesCng results can be found in the Rdata file:
22
> library("irace”) > testing.main(logFile=“exec-‐dir/irace.Rdata”)
> library("irace”) > load(“exec-‐dir/irace.Rdata”) > iraceResults$testing
irace tutorial – MatheurisCcs 2016
QuesCons?
Thank you for your aoenCon!
