Slide 1

Transformation of Input Space using Statistical Moments: EA-Based Approach Ahmed Kattan: Um Al Qura University, Saudi Arabia Michael Kampouridis: University of Kent, UK Yew-Soon Ong: Nanyang Technological University, Singapore Khalid Mehamdi: Um Al Qura University, Saudi Arabia

Slide 2

The problem Standard Regression models are presented with Observational data of the form (x i, y i ) i=1n Each x i denotes a k-dimensional input vector of design variables and y is the response. When k n, high variance and over-fitting become a major concern.

Slide 3

The problem High dimensional regression problem Regression Model Poor approximation

Slide 4

Solutions Curse of dimensionality is solved by: RReduce number of dimensions by selecting important features (e.g., PCA, FDA,..etc.) TTransformation of input space (e.g., GP, FFX,..etc.) Majority of work in this topic has been done for classification problems. The idea of transforming input space to reduce the number of design variables in the regression problems to improve generalisation is relatively little explored thus far.

Slide 5

Contributions of this work Contributions A novel evolutionary approach to transform the high-dimensional input space of regression models using only statistical moments. analysis to understand the impact of different statistical moments on the evolved transformation procedure dramatically improve LRs generalisation and make it competitive to other state-of-the-art regression models.

Slide 6

The proposed transformation (x i, y i ) (z i, y i ) Transformation x1x1,,, xkxk x0x0 z1z1 znzn z0z0 We transform the input vector x into and vector called z. The z is smaller than x and easier to be approximated by standard regression models.

Slide 7

The proposed transformation We used standard Genetic Algorithm

Slide 8

Genetic Algorithm Population representation

Slide 9

Genetic Algorithm Search operators Crossover in which two individuals exchange statistical moments and their parameters, randomly. op 0 op 1 op 2 op g a0a2a3a7a5a8a0a2a3a7a5a8 a 2 a 3 a 4 a 2 a 7... a0a2a7a0a2a7 a0a5a6a7a9a0a5a6a7a9 . op 0 op 1 op 2 op g a0a2a3a7a5a8a0a2a3a7a5a8 a 2 a 3 a 4 a 2 a 7... a0a2a7a0a2a7 a0a5a6a7a9a0a5a6a7a9 .

Slide 10

Genetic Algorithm Search operators Aggressive mutation operator that replaces a statistical moment and its parameters, randomly selected, with another randomly selected moments from the pool of statistical moments. op 1 op 2 op g a0a2a3a7a5a8a0a2a3a7a5a8 a 2 a 3 a 4 a 2 a 7... a0a2a7a0a2a7 a0a5a6a7a9a0a5a6a7a9 . a4a3a9a4a3a9 op 0 New op 0

Slide 11

Genetic Algorithm Search operators Smooth mutation operator where a parameter of a randomly selected statistical moment is mutated into a new parameter. op 0 op 1 op 2 op g a0a2a3a7a5a8a0a2a3a7a5a8 a 2 a 3 a 4 a 2 a 7... a0a2a7a0a2a7 a0a5a6a7a9a0a5a6a7a9 . a4a4

Slide 12

Genetic Algorithm Fitness measure We used average prediction errors of Linear Regression (LR) as a fitness measure for GA. LR is a very simple algorithm where it considers the family of linear hypotheses:

Slide 13

Genetic Algorithm Fitness measure Why LR ? Hence, given these features LR can push the GAs evolutionary process to linearly align the transformed inputs with their outputs and minimise the dimensionality of the new space.

Slide 14

Genetic Algorithm Fitness measure The GA aims to minimise the following fitness function:

Slide 15

Genetic Algorithm Training Two disjoint sets: training and validation. LR: two-folds cross-validation approach. The best individual in each generation is further tested with the validation set. We select the individual that yields the best performance on the validation set across the run.

Slide 16

Empirical tests We tested the effects of the transformation procedure on LR and compared the results against five regression models, namely: 1.RBFN 2.RBFN + PCA 3.Kriging 4.Kriging + PCA 5.LR 6.LR + PCA 7. piecewise LR 8.Genetic Programming 9.Genetic Programming + PCA

Slide 17

Empirical tests F1 = Rastrigin functionF2 = Schwefel function We tested 5 benchmark functions

Slide 18

Empirical tests F5 = Dixon & Price function F3 = Michalewicz function F4 = Sphere function

Slide 19

Empirical tests For each test function, we trained all regression models to approximate the given function when the number of variables is 100 variables. 500 variables. 1000 variables.

Slide 20

Empirical tests

Slide 21

Approximation Quality Sphere function for 2 variables

Slide 22

Empirical tests LR approximate the Sphere function after input transformation

Slide 23

Learn from evolution

Slide 24

It is clear from the heat maps that each problem has its unique characteristics. Interestingly, there is a consensus among all maps that the following operators do not contribute to the construction of good transformation procedures. copy copy intercept.

Slide 25

Learn from evolution Also, all maps agree that the following are important across all problems. Average Deviation Geometric Mean Min Max We still do not have a full understanding of the effect of these moments on the transformed space. In future research we will focus on this aspect.

Slide 26

Conclusions In this work we presented: A novel evolutionary approach to transform the high-dimensional input space of regression models using only statistical moments. analysis to understand the impact of different statistical moments on the evolved transformation procedure. dramatically improve LRs generalisation and make it competitive to other state-of-the-art regression models. We hope our results will inspire other researchers to build a deeper understanding to discover relations between straight statistical momnets on making good transformation

Slide 27

Thank you for paying attention!