1

Capabilities and Prospects of Inductive Modeling

Volodymyr STEPASHKO

Prof., Dr. Sci., Head of Department INFORMATION TECHNOLOGIES FOR INDUCTIVE MODELING

International Research and Training Centreof the Academy of Sciences of Ukraine

2

Layout 1. Historical aspects of IM2. International events on IM3. Attempt to define IM: what is it?4. IM destination: what is this for?5. IM explanation: basic algorithms and tools6. Basic Theoretical Results7. IM compared to ANN and CI8. Real-world applications of IM9. Main centers of IM research10. IM development prospects

3

1. Historical aspects of IM1968 First publication on GMDH:

Iвахненко O.Г. Метод групового урахування аргументів– конкурент методу стохастичної апроксимації // Автоматика. – 1968. – № 3. – С. 58-72.

Terminology evolution:heuristic self-organization of models (1970s) inductive method of model building (1980s) inductive learning algorithms for modeling (1992) inductive modeling (1998)

GMDH: Group Method of Data HandlingMGUA: Method of Group Using of Arguments

4

A.G.Ivakhnenko: GMDH originator

5

Main scientific results in inductive modelling theory:

Foundations of cybernetic forecasting device constructionTheory of models self-organization by experimental dataGroup method of data handling (GMDH) for automatic

construction (self-organization) of model for complex systemsMethod of control with optimization of forecastPrinciples of noise-immunity modelling from noisy dataPrinciples of polynomial networks constructionPrinciple of neural networks construction with active neurons

6

Academician Ivakhnenko•Originator of the scientific school of inductive modelling•Author of 44 monographs and numerous articls•Prepared more than 200 Cand. Sci (Ph.D.) and 27 Doct. Sci

Academician Ivakhnenko•Originator of the scientific school of inductive modelling•Author of 44 monographs and numerous articls•Prepared more than 200 Cand. Sci (Ph.D.) and 27 Doct. Sci

7

2. International events on IM2002 Lviv, Ukraine1st International Conference on Inductive Modelling

ICIM’20022005 Kyiv, Ukraine1st International Workshop on Inductive Modelling IWIM’20022007 Prague, Czech Republic2nd International Workshop on Inductive Modelling IWIM’20072008 Kyiv, Ukraine2nd International Conference on Inductive ModellingICIM’20082009 Krynica, Poland3rd International Workshop on Inductive Modelling IWIM’20092010 Yevpatoria, Crimea, Ukraine3rd International Conference on Inductive Modelling ICIM’2009Zhukyn (near Kyiv, Ukraine)Annual International Summer School on Inductive Modelling

8

3. Attempt to define IM: what is it?

IM is MGUA / GMDHIM is a technique for model self-organizationIM is a technology for building models from noisy

dataIM is the technology of inductive transition from

data to models under uncertainty conditions:small volume of noisy dataunknown character and level of noise inexact composition of relevant arguments (factors)unknown structure of relationships in an object

9

4. IM destination: what is this for?

IM is used for solving the following problems:Modelling from experimental dataForecasting of complex processesStructure and parametric identificationClassification and pattern recognitionData clasterizationMachine learningData MiningKnowledge Discovery

10

Given: data sample of n observations after m input x1, x2,…, xm and output y variables Find: model y = f(x1, x2,…, xm ,θ) with minimum variance of prediction error

GMDH Task: models ofset criterionquality model )(),(minarg , −ℑ−=ℑ∈

∗ fCfCff

Basic principles of the GMDH as an inductive method:

1. generation of variants of the gradually complicated structures of models

2. successive selection of the best variants using the freedom of decisions choice 3. external addition (due to the sample division) as the selection criterion

Part А Generation of models ℑ∈f

Calculation of criterion С( f ) C→ minSample

Part В f *

5. IM explanation: algorithms and tools

Basic Principles of GMDH as an Inductive Method

11

DD AA TT AA (( ss aa mm pp llee ,, aa pp rr iioo rr yy iinn ffoo rr mm aa tt iioo nn ))

CC hh oo iicc ee oo ff aa mm oo dd ee ll cc llaa ss ss SS tt rr uu cc ttuu rr ee gg ee nn ee rr aa tt iioo nn

PP aa rr aa mm ee ttee rr ee ss tt iimm aa tt iioo nn

CC rr ii ttee rr iioo nn mm iinn iimm iizz aa tt iioo nn AA dd ee qq uu aa cc yy aa nn aa llyy ss iiss FF iinn iiss hh iinn gg tthh ee pp rr oo cc ee ss ss

Main stages of the modeling process

12

GMDH featuresModel Classes: linear, polynomial, autoregressive, difference (dynamic), nonlinear of network type etc. Parameter estimation: Least Squares Method (LSM)Model structure generators:

GMDH Generators

Sorting-out Iterative

Exhaustive search

Directed search

Multilayered Relaxative

13

Main generators of models structures1. Combinatorial:

11 ,1,,,1,)|( −− === ssjs

is

ls FlimsxXy θ

))

2. Combinatorial-selective:

3. Selective (multilayered iterative):

2

25

24321

1

,1;,1,,...;1,0

,)()(

F

rj

ri

rj

ril

rjl

ril

rl

ClFjir

yyyyyyy

===

++++=+ ϑϑϑϑϑ

),...,,(;2...,,1, 21 mm

vvv ddddvXy === θ)

14

External Selection Criteria

Given sample: W = (X |y), X [nxm], y [nx1]Division into two subsamples:

nnnyy

yXX

XWW

W BAB

A

B

A

B

A =+⎥⎦

⎤⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡= ;;;

,,,,)( 1 WBAGyXXX GTGG

TGG == −θ

)Parameter estimation for a model y=Xθ:

Regularity criterion:2

BWAW XXCB θθ))

−=Unbiasedness criterion:

2ABBB XyAR θ)

−=

15

IM tools

Information Technology ASTRID (Kyiv)

KnowledgeMiner (Frank Lemke, Berlin)

FAKE GAME (Pavel Kordik at al., Prague)

GMDHshell (Oleksiy Koshulko, Kyiv)

16

6. Basic Theoretical Results

).(minarg* fCfFf∈

=

F – set of model structuresС – criterion of a model qualityStructure of a model:

)

Q – criterion of the quality of modelparameters estimation

),( ff Xfy θ) =Estimation of parameters:

).(minarg fR

f Qm

f

θθθ ∈

=)

17

Main concept:Self-organizing evolution of the model of

optimal complexity under uncertainty conditionsMain result:

Complexity of the optimum forecasting model depends on the level of uncertainty in the data: the higher it is, the simpler (more robust) there must be the optimum model Main conclusion:

GMDH is the method for construction of models with minimum variance of forecasting error

18

0

1

2

3

4

5

6

0 1 2 3 4 s

Jb(s)==J(s|0)

σ2 = 1,0

σ2 = 0

σ2 = 0,5

σ2 = 1,5 σ2 = 2,0 J(s|σ2)

0

1

2

3

4

5

6

0 0,5 1 1,5 2 2,5

s= 0

s= 1

σ2

s= 2s= 3s= 4J(σ2|s)

σ2кр(2,3) σ2кр(1,2) σ2кр(0,1)

Illustration to the GMDH theory

19

1x

2x

3x

4x

m

1f

2f

3f

4f

FCm ⇒2

1g

3g

4g

FCF ⇒2

2g

∗f

7. IM compared to ANN and CISelective (multilayered) GMDH algorithm:

20

1x

2x

3x

4x

m

1f

3f

4f

FCm ⇒2

4g

FCF ⇒2

2g

∗f

Optimal structure of the multilayered net

21

8. Real-world applications of IM1. Prediction of tax revenues and inflation2. Modelling of ecological processes

activity of microorganisms in soil under influence of heavy metalsirrigation of trees by processed wastewaterswater ecology

3. System prediction of power indicators4. Integral evaluation of the state of the complex

multidimensional systemseconomic safetyinvestment activityecological state of water reservoirspower safety

5. Technology of informative-analytical support of operative management decisions

22

9. Main centers of IM research

IRTC ITS NANU, Kyiv, UkraineNTUU “KPI”, Kyiv, UkraineKnowledgeMiner, Berlin, GermanyCTU in Prague, Czech Sichuan University, Chengdu, China

23

10. IM development prospects

The most promising directions:1. Theoretical investigations2. Integration of best developments of

IM, NN and CI3. Paralleling4. Preprocessing5. Ensembling6. Intellectual interface7. Case studes

24

THANK YOU!

Volodymyr STEPASHKO

Address: Prof. Volodymyr Stepashko, International Centre of ITS,

Akademik Glushkov Prospekt 40, Kyiv, MSP, 03680, Ukraine.

Phone: +38 (044) 526-30-28 Fax: +38 (044) 526-15-70

E-mail: stepashko@irtc.org.uaWeb: www.mgua.irtc.org.ua