Abstract Nature frequently serves as an inspirationfor developing new algorithms to solve challengingreal-world problems.Mathematicalmodeling has led tothe development of artificial neural networks (ANNs),which have proven especially useful for solving prob-lems such as classification and regression. Moreover,evolutionary algorithms (EAs), inspired by Darwin’snatural evolution, have been successfully applied tosolve optimization, modeling, and simulation prob-lems. Differential evolution (DE) is a particularly well-known EA that possesses a multitude of strategies forgenerating an offspring solution, where the best strat-egy is not known in advance. In this paper, the ANNregression is applied as a local search heuristic withinthe DE algorithm that tries predicting the best strategyor attempting to generate a better offspring from anensemble of DE strategies. This local search heuristic
I. Fister · I. Fister Jr. · D. StrnadFaculty of Electrical Engineering and Computer Science,University of Maribor, Smetanova 17, 2000 Maribor,Slovenia
P. N. SuganthanSchool of Electrical and Electronic Engineering, NanyangTechnological University, Singapore 639798, Singapore
S. M. Kamal · F. M. Al-Marzouki · M. PercDepartment of Physics, Faculty of Sciences, KingAbdulaziz University, Jeddah, Saudi Arabia
M. Perc (B)Faculty of Natural Sciences and Mathematics, Universityof Maribor, Koroška cesta 160, 2000 Maribor, Sloveniae-mail: [email protected]; [email protected]
is applied to the population of solutions according to acontrol parameter that regulates between the time com-plexity of calculation and the quality of the solution.The experiments on a CEC 2014 test suite consistingof 30 benchmark functions reveal the full potential indeveloping this idea.
Scientists in various research areas that are confrontedwith solving tough real-world problems have alwayssearched for an inspiration in the nature. Nature notonly poses the questions, but also provides answers tothese. In computer sciences, two nature-inspiredmech-anisms have been widely used: human brains [46] andDarwinian theory of struggle for survival [7]. The for-mer inspiration from the nature has led to the emer-gence of artificial neural networks (ANNs), while thelatter to evolutionary algorithms (EAs). In this paper,ANN is used as a regression method to enhance theperformance of differential evolution (DE).
Operation of an ANN simulates the electrochemi-cal activity of brain cells called neurons [46]. The firstmathematicalmodel of neuronswas devised byMcCul-loch and Pitts [37]. According to their model, a neuron“fires,” when a linear combination of weighted inputs
exceeds some threshold. Nonlinear response charac-teristic of a neuron is usually achieved through a sig-moid transfer function, which transforms the activationvalue to neuron output. In the most widely used typeof ANNs, the neurons are organized in layers. The out-puts of one layer become the weighted inputs to thenext layer, with no interconnections of neurons withinthe layer. One of primary practical uses of ANNs is toperform nonlinear regression on a set of input–outputpairs.Network training is executed in a supervised fash-ion by introducing the inputs to the network, observingthe output error with respect to the target values, andadjusting the connection weights to improve the per-formance in the next round.
On the other hand, DE has become one of themost prominent EAs for solving tough real-world opti-mization problems. This population-based method wasintroduced by Storn and Price in 1995 [48]. Individu-als in the population representing the solution of theproblem to be solved are in a form of real-valued vec-tors that are subjected to the operators of crossover andmutation. Thus, a population of trial vectors is gener-ated that compete with their parents for survival. As aresult, when a fitness of the trial vector is better thanor equal to the fitness of its parent at the same indexposition in the population, the parent is replaced by thetrial (offspring) solution.
In order to further improve the DE algorithm, itsdevelopment has been conducted in several ways. Forexample, adapting and self-adapting DEs assume thatthe parameters as set at the start of the search processmay not be appropriate in later phases. Therefore,these parameters are encoded into the solution vec-tor and undergo operations of crossover and mutation.Examples of successfully applied adaptive and self-adaptive DE algorithms are jDE [4] and SaDE [42].Another kind of DE algorithms tries to improve theresults of the original DE algorithmby using ensemblesof parameters and mutation DE strategies [34,49,50].A complete survey of DE methods can be foundin [8,51].
Selection of proper DE mutation strategy is prob-lem specific. Furthermore, the best strategymay changewith the search progress in the same way as other DEparameters. The problem of adapting the DE mutationstrategy has previously been addressed in [33]. In thispaper, we propose the use of ANN to build an adap-tive regression model for the best DEmutation strategyfrom an ensemble of DE strategies.
In [14], various DE strategies are applied for eachindividual, where the best value of element obtainedby all strategies in the DE ensemble is used to predictthe best value of the corresponding trial vector. Thiscontribution tries to overcome the time complexity ofthe ANN regression applied to each individual. Here,the ANN regression is used as a local search heuristicapplied to a candidate solution according to the controlparameter called probability of regression. The higherthe value of this parameter, the more candidate solu-tions undergo ANN local search heuristic.
The structure of the remainder of the paper is asfollows: In Sect. 2, we give a short overview of ANNand DE. Section 3 proposes a new DE algorithm withANN-based regression of DE strategies (nnDE). Theexperiments and the results are presented in Sect. 4.The paper concludes with a review of the paper contri-butions and prospects for future work.
2 Background
2.1 Artificial neural networks
An ANN is a mathematical model of human brain.It consists of a set of interconnected artificial neu-rons, which simulate the operation of natural neu-rons (i.e., brain cells) [45]. The electrochemical sig-nals in the brain are amplified and propagated alongneuronal chains, whereby each neuron receives a num-ber of input signals through ramified sensors (den-drites) and forwards an output signal through its singleextension (axon) [21]. The simplest artificial neuronmodel is shown in Fig. 1a. It receives the input vec-tor x = (x0, . . . , xn) and produces output y using thefollowing equation [37]:
y = φ
(n∑
i=0
wi xi
)(1)
The signal transfer function is programmable througha set w = (w0, . . . , wn) of weights on the input lines,where line 0 serves as an interceptwithfixed input valuex0 = −1. A common choice for the transfer function φ
is a Heaviside function or a sigmoid function like tanh.With respect to the variety of connectivity types that
emerge in different functional areas of the brain, manydifferent neural network architectures have been pro-posed in the past. The one that received thewidest prac-
123
Artificial neural network regression as a local search heuristic 897
Fig. 1 An example of an artificial neuron (a), and an artificial neural network (b). a Artificial neuron, b Multilayer perception
tical application and is also employed in the context ofour paper is a feedforward multilayer ANN that con-sists of neurons organized in several layers (Fig. 1b).Numerical input signals enter the network on the leftand propagate through layers toward the outputs onthe right. The interlayer signaling works by using theoutputs of layer i as inputs of layer i + 1. An estab-lished term for such type of ANN is a multilayer per-ceptron (MLP). All layers except the last output layerare referred to as hidden layers.
The spectrum of application domains for the ANNsis wide and includes, among others, applications inindustry and business [54], data mining [3], civil engi-neering [27], and fire safety [25]. In the field ofmachinelearning, the ANNs are used in classification andregression tasks [28,30,56].
In its role as nonlinear regressor, the MLP must betrained using a set of training samples with knowntarget values. This approach is known as supervisedlearning [23]. The training procedure is an iterativeprocess in which the networkweights are progressivelyadjusted such that the discrepancy between the networkoutputs and target values is minimized. The commonerror measures for network prediction quality are themean-squared error (MSE) and the cross-entropy error(CEE). The best known supervised learning algorithmfor MLP is the back-propagation method, which usesthe gradient of the error function to adapt the weightvectors. Weight changes are usually performed afterthe presentation of each individual training pattern andthe determination of output error, but can be delayed toafter the completion of a cycle of presentations (calledan epoch). The termination criterion for the training isdetermined by the allowed error tolerance, maximumnumber of training epochs, or one of themore advanced
methods like cross-validation of prediction efficiencyon a separate test set.
In this paper, we propose the use of a two-layerMLP as an aggregator for an ensemble of DE strate-gies, where the best member of current population isthe regression target and the trial vectors derived bydifferent DE strategies are used as training inputs.
2.2 Differential evolution
Differential evolution (DE) belongs to the class of evo-lutionary algorithms and is appropriate for solving con-tinuous as well as discrete optimization problems. DEwas introduced by Storn and Price in 1995 [48] andsince then many DE variants have been proposed. Theoriginal DE algorithm is represented by real-valuedvectors and is population-based. TheDE supports oper-ators, such as mutation, crossover, and selection.
In the basic mutation, two solutions are randomlyselected and their scaled difference is added to the thirdsolution, as follows:
u(t)i =x(t)
r0 +F · (x(t)r1 −x(t)
r2 ), for i =1 . . .NP, (2)
where F ∈ [0.1, 1.0] denotes the scaling factor thatscales the rate ofmodification,while r0, r1, r2 are ran-domly selected values in the interval 1 . . .NP and NPrepresents the population size. Note that the proposedinterval of values for parameter F was enforced in theDE community, although Price and Storn proposed theslightly different interval, i.e., F ∈ [0.0, 2.0].
DE employs a binomial (bin) or exponential (exp)crossover. The trial vector is built from parameter val-ues copied from either the mutant vector generated byEq. (2) or parent at the same index position i . Mathe-matically, this crossover can be expressed as follows:
123
898 I. Fister et al.
Table 1 An ensemble of DE strategies (ES)
No. Strategy Mutation expression Crossover
1 Best/1/Exp x (t+1)i, j = best(t)j + F · (x (t)
r1, j − x (t)r2, j ) Exponential
2 Rand/1/Exp x (t+1)i, j = x (t)
r1, j + F · (x (t)r2, j − x (t)
r3, j ) Exponential
3 RandToBest/1/Exp x (t+1)i, j = x (t)
i, j + F · (best(t)i − x (t)i, j ) + F · (x (t)
r1, j − x (t)r2, j ) Exponential
4 Best/2/Exp x (t+1)i, j = best(t)i + F · (x (t)
r1,i + x (t)r2,i − x (t)
r3,i − x (t)r4,i ) Exponential
5 Rand/2/Exp x (t+1)i, j = x (t)
r1,i + F · (x (t)r2,i + x (t)
r3,i − x (t)r4,i − x (t)
r5,i ) Exponential
6 Best/1/Bin x (t+1)i, j = best(t)i + F · (x (t)
r1,i − x (t)r2,i ) Binomial
7 Rand/1/Bin x (t+1)i, j = x (t)
r1, j + F · (x (t)r2, j − x (t)
r3, j ) Binomial
8 RandToBest/1/Bin x (t+1)i, j = x (t)
i, j + F · (best(t)i − x (t)i, j ) + F · (x (t)
r1, j − x (t)r2, j ) Binomial
9 Best/2/Bin x (t+1)i, j = best(t)i + F · (x (t)
r1,i + x (t)r2,i − x (t)
r3,i − x (t)r4,i ) Binomial
10 Rand/2/Bin x (t+1)i, j = x (t)
r1,i + F · (x (t)r2,i + x (t)
r3,i − x (t)r4,i − x (t)
r5,i ) Binomial
w(t)i, j =
{u(t)i, j rand j (0, 1) ≤ CR ∨ j = jrand,
x (t)i, j otherwise,
(3)
where CR ∈ [0.0, 1.0] controls the fraction of parame-ters that are copied to the trial solution. The conditionj = jrand ensures that the trial vector differs from theoriginal solution x(t)
i in at least one element.Mathematically, the selection can be expressed as
follows:
x(t+1)i =
{w(t)i if f (w(t)
i ) ≤ f (x(t)i ),
x(t)i otherwise .
(4)
Crossover and mutation can be performed in sev-eral ways in differential evolution. Therefore, a spe-cific notation was introduced to describe the varietiesof thesemethods (also strategies), in general. For exam-ple, rand/1/bin denotes that the base vector is randomlyselected, 1 vector difference is added to it, and thenumber of modified parameters in the trial/offspringvector follows a binomial distribution. The other stan-dard DE strategies are illustrated in Table 1. Thesestrategies also form an ensemble of DE strategies(ES).
2.3 An evolution of DE algorithms
Since its introduction in 1995, many variants of DEalgorithm have been developed so far. Zhang et al. [55]combined differential evolution with particle swarmoptimization, and result was a new algorithm called
DEPSO. Fan and Lampinen [13] added a new trigono-metric mutation operator to DE in 2003. Lin et al. [31]developed co-evolutionary hybrid DE. Chakrabortyet al. [6] proposed an improved variant of originalDE/best/1 scheme by utilizing the concept of the localtopological neighborhood of each vector. Scheme triesto balance exploration and exploitation abilities of DE,without using additional FES. Qin et al. [42] proposeda differential evolution algorithm with strategy adapta-tion. In 2006, Brest [4] proposed the concept of self-adaptation of control parameters. A new algorithm jDEwas proposed in [4]. Rahnamayan et al. [43] incor-porated an opposition-based mechanism in DE, whileDas et al. [9] proposed DE using neighborhood-basedmutation operator. Piotrowski [41] combined somewell-known DE approaches and gathered together inone framework as a new adaptive memetic DE withglobal and local neighborhood-based mutation oper-ators. Han et al. [22] created dynamic group-baseddifferential evolution using a self-adaptive strategy tocope with global optimization problems, while Cai andWang [5] developed differential evolution with neigh-borhood and direction information. Neri et al. [38–40]proposed compact differential evolution (cDE) whichcould run also on very limited hardware.
Differential evolution has been used to solve practi-cal problems such as electromagnetics [44], economicemission load dispatch problems [2], crop planningmodel [1], unit commitment problem [10], short-termelectrical power generation scheduling [52], ANNsdesign [20], protein structure prediction [47] and manymore.
123
Artificial neural network regression as a local search heuristic 899
2.3.1 Ensemble DE methods
In the literature, some ensemble DEmethods were pro-posed. Mallipeddi et al. [34–36] proposed an EPSDEalgorithm (differential evolution algorithmwith ensem-ble of parameters and mutation strategies) where apool of distinct mutation strategies along with a poolof values for each control parameter coexists through-out the evolution process and competes to produce off-spring. Mallipeddi and Suganthan [32] also proposeddifferential evolution algorithm with ensemble of pop-ulations to deal with global numerical optimization.Fister et al. [17] applied ensemble of DE strategiesto the hybrid bat algorithm [18]. Elsayed et al. [12]introduced an algorithm framework that uses multi-ple search operators in each generation. An appropri-ate mix of search operators is determined adaptively.LaTorre [29] explored the use of a hybrid memeticalgorithm based on the multiple offspring framework.Their algorithm combines the explorative/exploitativestrength of two heuristic searchmethods that separatelyobtain very competitive results. Vrugt et al. [53] pro-posed a concept, where different search algorithms runconcurrently and learn from each other through infor-mation exchange using a common population.
2.3.2 jDE algorithm
In 2006,Brest et al. [4] proposed an effectiveDEvariant(jDE),where control parameters are changed during therun. In this case, two parameters, namely scale factorF and crossover rate CR, are changed during the run.In jDE, every individual is extended with F and CR:
x (t)i = (x (t)
i,1, x(t)i,2, . . . , x
(t)i,D, F (t)
i ,CR(t)i ).
jDE are changing parameters F and CR accordingto the following equations:
F (t+1)i =
{Fl + rand1 ∗ (Fu − Fl) if rand2 < τ1,
F (t)i otherwise,
(5)
CR(t+1)i =
{rand3 if rand4 < τ2,
CR(t)i otherwise,
(6)
where randi=1...4 ∈ [0, 1] are randomly generated val-ues, τ1 and τ2 are learning steps, Fl and Fu lower andupper bound for parameter F .
3 The proposed algorithm
The proposed ANN regression on ensemble of DEstrategies (nnDE) (pseudo-code in Algorithm 1) mod-ifies the generation of the trial vector in the orig-inal DE algorithm. The trial vector yi is producedfrom the original vector xi by the default DE muta-tion strategy. A local search step (lines 5–10 in Algo-rithm 1) is then performed with probability pr . Theregression probability has a great impact on the perfor-mance of the algorithm, because it controls the num-ber of local search steps to be launched. Therefore,it influences the exploration and exploitation of theevolutionary search process. The higher the proba-bility of the regression, the more local search stepsare initiated. On the other hand, each local searchdemands an additional processor time that may causea performance degradation of the algorithm. As aresult, the proper value of this parameter needs tobe found for each specific problem on a case-by-casebasis.
During each local search, a regression ANN istrained using a training set Ti = {(t(k)i , xbest); k =1, . . . , P}, where each training pattern consists of aninput vector t(k)i of dimension D and a vector of net-work target outputs, which is xbest in all cases. Inputvectors t(k)i are derived from the currently processedpopulation member xi using randomly selected DEstrategies from the ensemble of strategies (ES) col-lected in Table 1. The neural network thus has Dinputs, 1 + log2 D hidden neurons, and D output neu-rons.
A trained ANN performs the regression of the bestfound solution from the set of trial solutions providedby the ensemble of DE strategies. When the strategiesagree and the trial solutions are similar, the nonlineartransformation represented by the trained ANN per-forms a narrowly directed local search. When this isnot the case, a random search ensues.
After training, the regression vector ri is obtainedby introducing the trial vector yi to the network. Theregression vector replaces the trial and is used in itsplace for subsequent fitness comparison with the orig-inal vector xi . Note that only one fitness evaluation isspent during this local search step because the genera-tion of the regression vector is performed in genotypicand not in phenotypic search space.
123
900 I. Fister et al.
Algorithm 1 The nnDE algorithm1: Initialize the DE population xi = (xi1, ..., xiD) for i =
1 . . .NP where NP is the population size.2: repeat3: for i = 1 to N P do4: Generate trial vectoryi fromxi usingdefaultDEstrategy;
5: if rand() < pr then6: Create a training set Ti from vector xi using ES and
target vector xbest;7: Train theANNusingTi till the number of epochs epoch
exceeded;8: Build regression vector ri from yi using the trained
ANN;9: yi = ri ;10: end if11: if f (yi ) < f (xi ) then12: xi = yi ;13: if f (yi ) < f (xbest) then14: xbest = yi ;15: end if16: end if17: end for18: until DE termination condition is met
In general, the efficiency of the local search dependson two facts [24]: How often it is launched and howextensively the local search process explores the searchspace. The former is controlled with the parameterpr , while the latter depends on the number of epochsneeded for training the ANN. Typically, a designerneeds to choose between the often launched short-termand rarely launched long-term local search. The short-term local search demands a smaller, while the long-term a larger number of ANN training epochs. In thisstudy, a rarely launched long-term local search heuris-tic is tested.
4 Experimental results
The goal of our experimental work is to show that usingthe ANN-based regression within the DE (nnDE) andself-adaptive jDE [4] (nnjDE) can improve the resultsof the original DE and jDE algorithms. In line withthis, the comparative study of thementioned algorithmswas performed by solving the CEC 2014 test suite. TheSaDE [42] method was also included in this compar-ative study. In order to analyze the impact of ANNregression on the original DE and jDE algorithms, thefollowing issues were investigated:
– the impact of the regression probability pr and thenumber of ANN training epochs epoch,
– the impact of the fitness function evaluations, and– the impact of problem dimensionality.
A comparative efficiency study of methods with andwithout the use of ANN local search was performed,and their convergence graphs were analyzed. The con-trol parameters of the DE and nnDE algorithms dur-ing the test were set as follows: F = 0.5,CR = 0.9,and NP = 100. The population size parameter NPwas the same for all compared algorithms. The propervalue of this parameter mainly depends on the prob-lem to be solved and was determined after extensiveexperimentation. The jDE and nnjDE algorithms’ para-meters were set as follows: F ∈ [0.1, 1.0],CR ∈[0.0, 1.0], τ1 = τ2 = 0.1. As a termination condi-tion, the number of fitness function evaluations wasused, as specified in the CEC 2014 benchmark suite,i.e., Tmax = 10000 · D. Each function was optimized25 times. The ANN training in nnDE and nnjDE wasterminated at 1000 epochs or when the training MSEdropped below 10−6, whichever occurred first. TheANN implementation from the OpenCV library wasused that supports a back-propagation method of train-ing, which was used in our tests with the learning rateand momentum scale set to 0.5 and 0.1, respectively.
4.1 Test suite
The CEC 2014 test suite (Table 2) consists of 30 bench-mark functions that are divided into four classes:
Unimodal functions have a single global optimumandno local optimums.Unimodal functions in this suiteare non-separable and rotated. Multi-modal functionsare either separable or non-separable. In addition, theyare also rotated or/and shifted. To develop the hybridfunctions, the variables are randomly divided into somesubcomponents, and then, different basic functions areused for different subcomponents [26]. Compositionfunctions consist of a sum of two or more basic func-tions. In this suite, hybrid functions are used as the basicfunctions to construct composition functions. Char-acteristics of these hybrid and composition functionsdepend on the characteristics of the basic functions.
The functions of dimensions D = 10, D = 20,and D = 30 were used in our experiments. The
123
Artificial neural network regression as a local search heuristic 901
Table 2 Summary of theCEC’14 test functions
No. Functions F∗i = Fi (x∗)
Unimodal functions 1 Rotated high conditioned elliptic function 100
2 Rotated bent cigar function 200
3 Rotated discus function 300
Simple multimodal functions 4 Shifted and Rotated Rosenbrocks function 400
5 Shifted and rotated Ackleys function 500
6 Shifted and rotated Weierstrass function 600
7 Shifted and rotated Griewanks function 700
8 Shifted Rastrigins function 800
9 Shifted and rotated Rastrigins function 900
10 Shifted Schwefels function 1000
11 Shifted and rotated Schwefels function 1100
12 Shifted and rotated Katsuura function 1200
13 Shifted and rotated HappyCat function 1300
14 Shifted and rotated HGBat function 1400
15 Shifted and rotated expanded Griewanksplus Rosenbrocks function
1500
16 Shifted and rotated expanded Scaffers F6function
1600
Hybrid functions 17 Hybrid function 1 (N = 3) 1700
18 Hybrid function 2 (N = 3) 1800
19 Hybrid function 3 (N = 4) 1900
20 Hybrid function 4 (N = 4) 2000
21 Hybrid function 5 (N = 5) 2100
22 Hybrid function 6 (N = 5) 2200
Composition functions 23 Composition function 1 (N = 5) 2300
24 Composition function 2 (N = 3) 2400
25 Composition function 3 (N = 3) 2500
26 Composition function 4 (N = 5) 2600
27 Composition function 5 (N = 5) 2700
28 Composition function 6 (N = 5) 2800
29 Composition function 7 (N = 3) 2900
30 Composition function 8 (N = 3) 3000
search range of the problem variables is limited toxi ∈ [−100, 100].
4.2 Impacts of the regression probabilityand the number of ANN training epochs
The goal of this experiment is twofold. Firstly, we aimto discover how the parameter pr affects the resultsof the nnDE and nnjDE algorithms on the CEC 2014test suite, and secondly, we want to explore how thenumber of ANN training epochs influences the resultsof the nnDE algorithm on the same test suite.
In the first experiment, the probability of localsearch application was varied in the interval pr ∈[0.005, 0.05] in steps of 0.005, resulting in ten launchconfigurations per problem size D. The results of eachconfiguration according to five statistical measures(i.e., minimum, maximum, average, median, and stan-dard deviation) accumulated over 25 runs for each func-tion were aggregated into a statistical classifier consist-ing of 30 × 5 = 150 variables that served as inputto Friedman statistical test. The Friedman test [11]calculates the average method ranks over all prob-lem instances (i.e., benchmark functions) for each ofthe test configurations. For the case D = 10, Fig. 2
123
902 I. Fister et al.
Fig. 2 Average rank differences of nnDE (a) and nnjDE (b)algorithms achieved over all problem instances with D = 10 fordifferent settings of pr . The rank difference is expressed as bar
height and direction. As a result, all bars higher than zero indi-cate that the corresponding hybrid DE algorithm outperformedthe original DE algorithm in a particular parameter setting
illustrates the differences of average ranks achieved bynnDE and nnjDE in comparison with the original DEand jDEmethods, respectively. In the figure, each posi-tive average rank distancemeans that the correspondinginstance of nnDE and nnjDE outperformed the resultsof the original DE and jDE algorithms, and vice versa.Namely, negative average rank differences indicate thatthe original DE and jDE algorithms outperformed theresults achieved by the nnDE and nnjDE algorithms.
Two conclusions can be deduced from the exper-imental results. Firstly, the obtained results stronglydepend on the probability of regression pr . Secondly,the best results for D = 10 are obtained when pr =0.01, which complies with Piotrowski [41] who pro-posed performing the local search with probabilitypr = 0.005 when 100 · D fitness function evaluationsare spent per launch.
In the second experiment, the ANN training epochsin the nnDE algorithm were varied by epoch ∈{100, 500, 1000, 2000, 5000} for benchmark functionsof dimensions D = 10, D = 20, and D = 30. Exten-sive experiments showed that the setting pr = 0.01,where one local search step is launched in averagewhenusing the population size Np = 100, is not optimal. Itturns out that the optimal value of this parameter laysin a range pr ∈ (0.0, 0.01]. Therefore, it is varied aspr ∈ {0.01, 0.005, 0.003} in our tests, which corre-sponds to one call of the local search heuristic everyone, two, and three generations, respectively.
The average ranks obtained by Friedman nonpara-metric tests for experiment results obtained by nnDEwith different problem dimensions are illustrated inFig. 3. Each graph plots the average rank against thenumber ofANN training epochs. Each line correspondsto one of the tested values of pr .
Two facts can be concluded from the figure, as fol-lows:
– the smaller the probability of regression, the betterthe results,
– the higher the dimensionof the problem, the smallerthe number of epochs required.
in order to obtain the best results when optimizingthe benchmark functions of dimension D = 10, thenumber of ANN training epochs epoch = 2000 isneeded, while epoch = 100 is enough to obtain thebest results for dimension D = 30. The number ofANN training epochs depends on the probability ofregression by optimizing the functions of dimensionsD = 20, i.e., epoch ∈ {100, 500, 1000, 2000, 5000}and pr ∈ {0.01, 0.005, 0.003}.
Although infrequently executed, the local searchstep significantly influences the results of the optimiza-tion. On the other hand, theANN training starts to dom-inate the optimization runtime with a growing num-ber of training epochs. Fortunately, the smaller numberof required epochs and infrequent launching of localsearch steps make solving of the problem optimizationtractable in higher dimensions.
123
Artificial neural network regression as a local search heuristic 903
Fig. 3 Influence of the epoch number on the results of the optimization obtained by the nnDE, where each diagram consists of threecurves representing average ranks according to the specific probability of regression pr . a D = 10, b D = 20, c D = 30
4.3 Impact of the fitness function evaluations
One of the more reliable indicators of search stagna-tion is when the best result is not improved for a longterm. Alternatively, this can also mean that the searchprocess gets stuck in a local optimum. In order to detectthese undesirable situations during the run of nnDE, thefitness values were monitored at three different phasesof the search process, i.e., at 1/25, 1/5, and at the finalfitness function evaluation. The results of this test arecollated in Tables 3 and 4 for functions of dimensionD = 20.
As can be seen from Tables 3 and 4, nnjDE success-fully progressed toward the global optimum according
to all benchmark functions, i.e., no stagnation of thesearch process is detected.
4.4 Impact of the problem dimensionality
The goal of this experiment is to discover how thequality of the results obtained by the nnDE dependson the dimension of the problem. In line with this,three different dimensions of the benchmark functionsD ∈ {10, 20, 30} were taken into account. The resultsof the tests according to five measures are presented inTables 5 and 6.
In this experiment, it was expected that the functionsof the higher dimensions would be harder to optimize,
123
904 I. Fister et al.
Table 3 Results of the nnDE with pr = 0.003 and epoch = 2000 showing an impact of the fitness function evaluations measured after125 , 1
5 , and11 of the maximum fitness function evaluations for dimension D = 20—Part 1/2
Table 5 Results of the nnDE with pr = 0.003 and epoch ∈ {2000, 2000, 100} showing an impact of the dimensionality of the problemmeasured by function dimensions D ∈ {10, 20, 30}, respectively—Part 1/2
and therefore, the obtained results would be worse. Asamatter of fact, this assumption holds in general exceptfor functions f7, f11 and f12, where nnDE achievedbetter results by optimizing the functions of dimensionD = 20 than by the other dimensions. Interestingly,the results of optimizing the function f26 are equal bythe all algorithms in test.
4.5 A comparative study
In order to show that the hybridization with ANNregression improves the results of the original DE andjDE algorithms, a comparative study was performed.In this study, the results of the DE and jDE algo-rithms were compared with the results of nnDE andnnjDE, i.e., hybridized versions of the DE algorithmswith pr = 0.003 and epoch = 1000. All the men-tioned algorithms used the parameters as reported at
the beginning of this section. This comparative studywas widened by an additional self-adaptive DE algo-rithm, i.e., the SaDE using the following parameter set-tings: F is randomly selected from the normal distrib-ution N (0.5, 0.3), while CR is randomly drawn from anormal distribution N (CRmk, 0.1). The variableCRmk
denotes an average value of parameterCR for k-th strat-egy (four strategies were used in the ensemble strate-gies) during the last LP = 20 (i.e., learning rate para-meter) generations. The results according to the meanand standard deviation obtained by solving the CEC2014 benchmark functions of dimension D = 30 areillustrated in Table 7. The best results in the table arepresented in bold.
As it can be seen from Table 7, the nnjDE out-performs the results of the other observed algorithmstwelve times, SaDE eight times, nnDE and DE fourtimes, and jDE once. All algorithms achieved the sameresults for functions f23 and f26. Note that the nnDE
123
908 I. Fister et al.
Table 6 Results of the nnDE with pr = 0.003 and epoch ∈ {2000, 2000, 100} showing an impact of the dimensionality of the problemmeasured by function dimensions D ∈ {10, 20, 30}, respectively—Part 2/2
Fig. 4 Results of the Friedman nonparametric test, where eachdiagram illustrates the normalized average rank of the algorithmsin test for the specified dimensions of the benchmark functions.
The closer to one the value of the algorithm’s rank, the moresignificant is the specific algorithm. a D = 10, b D = 20, cD = 30
123
Artificial neural network regression as a local search heuristic 911
Fig. 5 Convergence graphs for six selected functions from the benchmark function suite. a f6 (D = 10), b f8 (D = 10), c f12 (D = 20),d f18 (D = 20), e f22 (D = 30), f f28 (D = 30)
123
912 I. Fister et al.
and SaDE obtained the same results for the function f9.However, the statistical analysis takes into account alsothe minimum, maximum, median, and standard devi-ation values. This comparison is therefore more accu-rate. In summary, the nnDE is thus better for solvingproblems of higher dimensions (i.e., D = 30), whilethe nnjDE is better for solving the problems of lowerdimensions (i.e., D = 10 and D = 20).
In order to evaluate the quality of the results statisti-cally, Friedman tests [19] were conducted that comparethe average ranks of the compared algorithms. Thus, anull hypothesis is placed that states: All algorithms areequivalent, and therefore, their ranks should be equal.When the null hypothesis is rejected, the Nemenyi posthoc test [11] is performed, where the critical differ-ence is calculated between the average ranks for eachalgorithm.
Three Friedman tests were performed regarding thevalues of fivemeasures obtained byoptimizing 30 func-tions of three different dimensions. As a result, eachalgorithm in the tests was compared with respect to150 variables. The tests were conducted at the signif-icance level 0.05. The results of the Friedman non-parametric test can be seen in Fig. 4, which is dividedinto three diagrams. Each diagram shows the ranks andconfidence intervals (critical differences) for the algo-rithms under consideration with regard to each prob-lem dimensionality. Note that a significant differencebetween two algorithms is observed if their confidenceintervals denoted as thickened lines in Fig. 4 do notoverlap.
Figure 4a–c shows that the original DE algorithmwas significantly outperformed by all other algorithmsin the test for all problem dimensions. The nnjDE algo-rithm exhibits the best results in dimensions D = 10(Fig. 4a) and D = 20 (Fig. 4b), while nnDE dom-inates the competitors for D = 30 (Fig. 4c). Asdemonstrated, the proposed local search heuristic sig-nificantly improves the results of both original DE andjDE algorithms, with the exception of the case nnjDEvs. jDE, where the advantage of nnjDE is not conclu-sive. Thereby, the assertion set at the beginning of thesection has been successfully confirmed.
4.6 Convergence analysis
Convergence graphs were analyzed for functions f6and f8 of dimension D = 10, functions f12 and f18
of dimension D = 20, and functions f22 and f28 ofdimension D = 30. The best out of 25 optimizationruns was analyzed. Convergence graphs are illustratedin Fig. 5 with two diagrams per problem dimension.
The following observations can be seen from thesegraphs:
– nnjDE outperforms the results of the original jDEfor optimization of all presented functions, exceptf28,
– nnDE outperforms the results of the original DE foroptimization of all presented functions, except f22and f28,
– all algorithms achieved the similar results for opti-mization of the function f28.
In summary, the presented results confirmed thathybridizing the original DE and jDE algorithms withthe ANN regression can improve the results of both.
5 Conclusion
Recently, hybridizing the nature-inspired algorithmsin order to expand its applicability and improve per-formance has become a popular trend in computa-tional intelligence [15,16]. This paper proposes thehybridization of a DE algorithm with an ANN-basedregression as a way to apply the local search heuristic.The ANN functions as a predictor of the best solu-tion from a training set of trial vectors produced by anensemble of DE strategies.
As a result, two hybrid DE algorithms were devel-oped, i.e., nnDE representing the hybridization of theoriginalDEalgorithmwithANN, and nnjDE represent-ing the hybridization of the original jDE algorithmwithANN. The results of experiments conducted on a CEC2014 test suite consisting of 30 benchmark functionshave shown that the proposed hybrids substantially out-perform their original predecessors. Moreover, the per-formances gap broadened when the dimensionality ofthe problem was increased.
The experiments suggest that the quality of resultshighly depends on the value of parameter pr , whichdetermines the probability of local search execution.Lower values of pr are generally required for higherproblem dimensions.
These preliminary results advocate further investi-gation of the proposed hybridization in the future. Asthe first next step, however, we would like to expand
123
Artificial neural network regression as a local search heuristic 913
our comparative study to other well-known EAs and SIalgorithms. Also, adaptation and self-adaptation of theparameter pr seem very promising idea for the future.
Acknowledgments This research was supported by theSlovenian Research Agency (Grant P5-0027) and by the Dean-ship of Scientific Research, King Abdulaziz University (Grant76-130-35-HiCi).
2. Bhattacharya, A., Chattopadhyay, P.K.: Solving economicemission load dispatch problems using hybrid differentialevolution. Appl. Soft Comput. 11(2), 2526–2537 (2011)
3. Bigus, J.P.: Data Mining with Neural Networks: SolvingBusiness Problems from Application Development to Deci-sion Support. McGraw-Hill, Inc., New York (1996)
4. Brest, J., Greiner, S., Boskovic, B., Mernik, M., Zumer, V.:Self-adapting control parameters in differential evolution: acomparative study on numerical benchmark problems. IEEETrans. Evolut. Comput. 10(6), 646–657 (2006)
5. Cai, Y., Wang, J.: Differential evolution with neighborhoodand direction information for numerical optimization. IEEETrans. Cybern. 43(6), 2202–2215 (2013)
6. Chakraborty,U.K.,Das, S., Konar,A.:Differential evolutionwith local neighborhood. In: IEEE Congress on Evolution-ary Computation, pp. 2042–2049. IEEE (2006)
7. Darwin, C.: On the Origin of Species. Harvard UniversityPress, London (1859)
9. Das, S., Abraham, A., Chakraborty, U.K., Konar, A.: Differ-ential evolution using a neighborhood-based mutation oper-ator. IEEE Trans. Evolut. Comput. 13(3), 526–553 (2009)
10. Datta, D., Dutta, S.: A binary-real-coded differential evo-lution for unit commitment problem. Int. J. Electr. PowerEnergy Syst. 42(1), 517–524 (2012)
11. Demšar, J.: Statistical comparisons of classifiers over mul-tiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)
13. Fan,H.Y., Lampinen, J.: A trigonometricmutation operationto differential evolution. J. Global Optim. 27(1), 105–129(2003)
14. Fister, I.J., Suganthan, P.N., Strnad, D., Brest, J., Fister, I.:Artificial neural network regression on ensemble strategiesin differential evolution. In: MENDEL 2014, 20th Interna-tional Conference on Soft Computing. University of Tech-nology, Brno (2014)
15. Fister, I., Fister Jr, I., Yang, X.S., Brest, J.: A comprehensivereviewof firefly algorithms. SwarmEvolut. Comput. 13, 34–46 (2013)
16. Fister, I., Rauter, S., Yang, X.S., Ljubic, K., Fister Jr, I.:Planning the sports training sessions with the bat algorithm.Neurocomputing 149(Part B), 993–1002 (2015)
17. Fister Jr, I., Fister, D., Fister, I.: Differential evolution strate-gies with random forest regression in the bat algorithm. In:Proceeding of the Fifteenth Annual Conference Compan-ion on Genetic and Evolutionary Computation ConferenceCompanion 2013, pp. 1703–1706. ACM (2013)
18. Fister Jr, I., Fister, D., Yang, X.S.: A hybrid bat algorithm.Elektroteh. vestnik 80(1–2), 1–7 (2013)
19. Friedman, M.: A comparison of alternative tests of signifi-cance for the problem of m rankings. Ann. Math. Stat. 11,86–92 (1940)
20. Garro, B.A., Sossa, H., Vázquez, R.A.: Design of artificialneural networks using differential evolution algorithm. In:Wong,K.W.,Mendis, B.S.U., Bouzerdoum,A. (eds.) NeuralInformation Processing. Models and Applications, pp. 201–208. Springer, Heidelberg (2010)
21. Gershenson, C.: Artificial Neural Networks for Beginners(2003). arXiv:cs/0308031
22. Han, M.F., Liao, S.H., Chang, J.Y., Lin, C.T.: Dynamicgroup-based differential evolution using a self-adaptivestrategy for global optimization problems. Appl. Intell.39(1), 41–56 (2013)
23. Hecht-Nielsen, R.: Theory of the backpropagation neuralnetwork. In: International Joint Conference on Neural Net-works, pp. 593–605. IEEE (1989)
24. Holger, H., Thomas, S.: Stochastic Local Search: Founda-tions & Applications. Morgan Kaufman Inc., Amsterdam(2004)
25. Hozjan, T., Turk, G., Srpcic, S.: Fire analysis of steel frameswith the use of artificial neural networks. J. Constr. SteelRes. 63(10), 1396–1403 (2007)
26. J.J. Liang, B.Y.Q., Suganthan, P.N.: Problem definitions andevaluation criteria for the CEC 2014 special session andcompetition on single objective real-parameter numericaloptimization. Technical Report, Zhengzhou University andNanyang Technological University (2013)
27. Kartam, N., Flood, I., Garrett, J.H.: Artificial Neural Net-works for Civil Engineers: Fundamentals and Applications.American Society of Civil Engineers, New York (1997)
28. Kourentzes, N., Barrow, D.K., Crone, S.F.: Neural networkensemble operators for time series forecasting. Expert Syst.Appl. 41(9), 4235–4244 (2014)
29. LaTorre, A., Muelas, S., Peña, J.M.: A mos-based dynamicmemetic differential evolution algorithm for continuousoptimization: a scalability test. Soft Comput. 15(11), 2187–2199 (2011)
30. Lee, S., Choeh, J.Y.: Predicting the helpfulness of onlinereviews usingmultilayer perceptron neural networks. ExpertSyst. Appl. 41(6), 3041–3046 (2014)
32. Mallipeddi, R., Suganthan, P.: Differential evolution algo-rithm with ensemble of populations for global numericaloptimization. Opsearch 46(2), 184–213 (2009)
33. Mallipeddi, R., Mallipeddi, S., Suganthan, P.N.: Ensemblestrategies with adaptive evolutionary programming. Inf. Sci.180(9), 1571–1581 (2010)
47. Santos, J., Diéguez, M.: Differential evolution for proteinstructure prediction using theHPmodel. In: Ferrández, J.M.,Álvarez, J.R., de la Paz, F., Toledo, F.J. (eds.) Foundations onNatural and Artificial Computation, pp. 323–333. Springer,Heidelberg (2011)
48. Storn, R., Price, K.: Differential evolution-a simple andefficient heuristic for global optimization over continuousspaces. J. Global Optim. 11(4), 341–359 (1997)
49. Tvrdìk, J.: Competitive differential evolution. In: MENDEL2006. 12th international conference on soft computing, pp.7–12. University of Technology, Brno (2006)
50. Tvrdìk, J.: Differential evolution with competitive setting ofits control parameters. TASK Q. 11, 169–179 (2007)
51. Tvrdìk, J.: Adaptation in differential evolution: a numericalcomparison. Appl. Soft Comput. 9, 1149–1155 (2009)
52. Uyar, A.S., Türkay, B., Keles, A.: A novel differential evo-lution application to short-term electrical power generationscheduling. Int. J. Electr. Power Energy Syst. 33(6), 1236–1242 (2011)
53. Vrugt, J.A., Robinson, B.A., Hyman, J.M.: Self-adaptivemultimethod search for global optimization in real-parameter spaces. IEEE Trans. Evolut. Comput. 13(2), 243–259 (2009)
54. Widrow, B., Rumelhart, D.E., Lehr, M.A.: Neural networks:applications in industry, business and science. Commun.ACM 37(3), 93–105 (1994)
![Convolutional neural network-based regression for …arXiv:1802.00664v1 [cs.CV] 2 Feb 2018 Convolutional neural network-based regression for depth prediction in digital holography](https://static.documents.pub/doc/80x56/5fc9b810f7f5f41d2e282d16/convolutional-neural-network-based-regression-for-arxiv180200664v1-cscv-2-feb.jpg)
