A Tensor Based Stacked Fuzzy Networks for Eｨcient Data Regression Jie Li Inner Mongolia University Jiale Hu Inner Mongolia University Guoliang Zhao ( [email protected]) Inner Mongolia University https://orcid.org/0000-0002-0323-1299 Sharina Huang Inner Mongolia Agriculture University: Inner Mongolia Agricultural University Yang Liu Inner Mongolia University Research Article Keywords: Extreme learning machine (ELM), Random vector functional link network (RVFL), Tensor-based type-2 random vector functional link network (T2-RVFL), Tensor-based type-2 Extreme learning machine (TT2-ELM), Tensor stacked fuzzy neural network (TSFNN) Posted Date: November 11th, 2021 DOI: https://doi.org/10.21203/rs.3.rs-1035440/v1 License: This work is licensed under a Creative Commons Attribution 4.0 International License. Read Full License
Transcript
A Tensor Based Stacked Fuzzy Networks forEcient Data RegressionJie Li
Random vector functional link and extreme learning machine have been extended by the type-2 fuzzy sets with vector stacked methods, this extension leads to a new way to use tensor toconstruct learning structure for the type-2 fuzzy sets-based learning framework. In this paper,type-2 fuzzy sets-based random vector functional link, type-2 fuzzy sets-based extreme learningmachine and Tikhonov-regularized extreme learning machine are fused into one network, a ten-sor way of stacking data is used to incorporate the nonlinear mappings when using type-2 fuzzysets. In this way, the network could learning the sub-structure by three sub-structures’ algo-rithms, which are merged into one tensor structure via the type-2 fuzzy mapping results. To thestacked single fuzzy neural network, the consequent part parameters learning are implementedby unfolding tensor-based matrix regression. The newly proposed stacked single fuzzy neural net-work shows a new way to design the hybrid fuzzy neural network with the higher order fuzzysets and higher order data structure. The effective of the proposed stacked single fuzzy neuralnetwork are verified by the classical testing benchmarks and several statistical testing methods.
Keywords: Extreme learning machine (ELM), Random vector functional link network (RVFL),Tensor-based type-2 random vector functional link network (T2-RVFL), Tensor-based type-2 Extremelearning machine (TT2-ELM), Tensor stacked fuzzy neural network (TSFNN)
1
Springer Nature 2021 LATEX template
Networks
1 Introduction
The random vector functional link (RVFL) andextreme learning machine (ELM) are two popu-lar randomized single layer forward learning net-works, which provide us a unified framework forboth regression and multi-class classification withsingle layer. Then the semi-supervised RVFL andELM networks can be merged into a joint opti-mization framework, it shows that the algorithmis efficient in moderate scale data classification(Peng et al, 2020). The parameters could be reg-ularized when ridge regression is used (Yildirimand Revan Ozkale, 2019). When singular valuedecomposition (SVD) is used for algorithm itera-tive solution searching, the SVD update algorithmscales better and works faster than SVD com-puted from scratch (Grigorievskiy et al, 2016).Multi-label learning method could also use themulti-label radial basis function neural networkand Laplacian ELM (Xu et al, 2019), in thisalgorithm, the clustering algorithm determinesthe number of hidden nodes, and the center ofthe activation function could be determined bythe data itself, then the output is solved by aLaplacian ELM. Inspired by biological intelligentsystems, bio-inspired learning model blooms a lotrecently (Huang and Chen, 2016; Alencar et al,2016; Christou et al, 2019), its can be applied tomany areas, such as, anomalous trajectory classi-fication (Sekh et al, 2020), long-term time seriesprediction (Grigorievskiy et al, 2014), T–S fuzzymodel identification (Wei et al, 2020), dictionarylearning-based image classification (Zeng et al,2020a), anomaly detection (Hashmi and Ahmad,2019), HRV recognition (Bugnon et al, 2020),energy system (Yaw et al, 2020), mislabeled sam-ples detection (Akusok et al, 2015), concept driftdetection (Yang et al, 2020c), etc.
Generalization performance is the main con-cern for the learning algorithms, balancing com-putational complexity and generalization abilityhave been extended via ELM (Ragusa et al,2020); with the designed data and modeled par-allel ELMs, large-scale learning tasks could betackled by ELM (Ming et al, 2018), moreover,a tradeoff should be made among efficiency andscalability, the algorithm should have complemen-tary advantages. With the aid of graph learningand adaptive unsupervised/semi-supervised clus-tering method, flexible and discriminative data
embedding could be achieved (Zeng et al, 2020b;Zheng et al, 2020). By using the regularized cor-rentropy criterion and half-quadratic optimizationtechnique, convergence speed and performance areboth showed superiorities than the original (Yanget al, 2020a), and the robust type algorithm hasbeen studied in (Yang et al, 2020b). When inverse-free recursive algorithm is used to update theinverse of the networks’ Hermitian matrix, effi-cient inverse-free algorithm is designed to updatethe regularized pseudo-inverse has been proposedfor ELMs (Zhu and Wu, 2020).
When big data environment is encountered,a fast parameter selection scheme for modelingthe large amount of data is needed, alternatingdirection method and maximally splitting methodcould be applied to the algorithms to minus thenumber of the sub-model and coefficients training(Lai et al, 2020). Concerning the credit probabilityfor network output for each prediction, probabilis-tic output from the original architecture of ELMis proposed, iterative way of learning is elimi-nated, and the merits of ELM is preserved (Wonget al, 2020). Using Bayesian inferences, multiple-instance learning-based ELM has proven to beefficient in classification problems (Wang et al,2020). Optimally pruned ELM (Miche et al, 2010)is presented to both regression and classificationproblems, the proposed algorithm could counterthe effect of noise. To move forward, a L2 regu-larization penalty applied to the optimally prunedELM, and a double-regularized ELM using LARSand Tikhonov regularization is proposed (Micheet al, 2011b). Missing data case for the regressionproblem is studied in (Yu et al, 2013). When train-ing sample selection method is designed based onthe fuzzy C-means clustering algorithm, and theproposed small training samples selection-basedhierarchical ELM could reduce the computationaltime (Xu et al, 2020).
Random vector functional link networks(RVFL) (Zhang and P.N.Suganthan, 2016) couldalso use the techniques mentioned above, suchas ridge regression (Zhang and Suganthan, 2017),and its extended version, the new learningparadigm is named as RVFL plus (RVFL+)(Zhang, 2020), it has been used in neuro-imaging-based parkinson’s disease diagnosis (Xue et al,2018; Shi et al, 2019). Motivated by the ELMs andRVFLs, generalized Moore-Penrose inverse andtriangular type-2 fuzzy sets are used to extend the
Springer Nature 2021 LATEX template
Networks 3
ELM, and tensor-based ELM has been proposed(Huang et al, 2019). RVFL network has been alsoexpanded to tensor case (Zhao and Wu, 2019), thetype-reduction method for general type-2 fuzzysets are removed in this type of network.
Motivated by the above-mentioned material,we have noticed that the type-2 fuzzy sets andtensor structure provide a new way to model thecomplex data, whether the ELMs, RVFLs or theneuro-fuzzy systems could be used under the ten-sor structure. Our target is to unveil the linksor laws behind the data, and a new tensor-basedstacked networks for efficient data regression isstudied. To get the merits of the algorithms,a good way is to fuse the algorithms into oneframe, the balance of performance and structuresimplicity would be achieved.
To fuse different concept and techniques intothe algorithm, it is inevitable to extract the differ-ent aspect of the data, then different view resultsof the data can be obtained, then the original pro-posed algorithms could be used to minimize thetesting error. Go back to the type-2 fuzzy sets,it could map the data with different parameter-specified fuzzy membership functions with at leastthree type-1 fuzzy membership functions, then themulti-view on the data could be obtained. A ques-tion follows this is how to fuse the results intoone data structure, tensor is the suitable choicefor these types of learning methods, this is themotivation of the work.
The structure of the rest of this article isas follows: Section 2 introduces the three algo-rithms that is used by stacking system, Section 2.1introduces the tensor-based type-2 RVFL, Section2.2 presents the tensor-based type-2 ELM, andSection 2.3 is the introduction to the TROP-ELM,it is used to compare algorithms’ performance. InSection 3, the structure of tensor based hybridsingle fuzzy neural networks, that is, a stackedsingle fuzzy neural network is presented. Simula-tion results and discussions are given in Section 4.Finally, conclusions are inferred in Section 5.
2 Preliminary
In this station, the tensor-based type-2 RVFL, ten-sor based type-2 ELM and Tikhonov regularizedOP-ELM is introduced.
2.1 Tensor-based type-2 RVFL
The RVFL usually adopts the activation func-tions to construct the network, for example theRadbas (y = e−s2) functions. In the hidden lay-ers of the network, where y and s are defined asthe outputs and inputs, respectively. The enhance-ment nodes of tensor-based RVFL are replacedwith IT2 fuzzy sets. The structure of tensor-based type-2 RVFL (TT2-RVFL) is representedin Fig. 1. Activation function Radbas of RVFL isextended to interval type-2 fuzzy set IT2Radbas,and the extended RVFL is constructed usingIT2Radbas. The Radbas activation function inRVFL is extended to interval type-2 fuzzy setsIT2Radbas, and IT2Radbas is used to constructIT2RVFL.
Output Layer
Input Layer
Fig. 1 Structure of the TT2-RVFL network.
Fig. 2 shows the standard formulation of theIT2 fuzzy set. The incentive function is namedIT2Radbas, and the interval type-2 fuzzy setcould be constructed with this type of activationfunctions.
0
1
2
3
4 0
0.5
1
1.5
20
0.5
1
Fig. 2 Structure of the it2 fuzzy set.
The membership function (MF) of type-2 fuzzysets in TT2-RVFL is defined as follows
gi(xi) = exp(−k1s2), (1)
gi(xi) = exp(−k2s
2), (2)
Springer Nature 2021 LATEX template
Networks
where s = xi − mi, k1 = 1σ2i
, k2 = 1σ2i
(i =
1, 2, · · · , L).Given testing data set Dt
Nt=1
, where Dt =(xt, yt), xt ∈ R
N and xt = (xt1, xt2, · · · , xtN ),yt ∈ R and y = [y1, · · ·, yN ]T . For a lower MFmatrix Φ ∈ R
N×L×1 can be structured with thefollowing R
N×L matrices as follows
Φ:,1,1 =
g1(a11x1 + b11) g
1(a12x1 + b12)
......
g1(a11xN + b11) g
1(a12xN + b12)
,
...
Φ:,L,1 =
gL(aL1x1 + bL1) g
L(aL2x1 + bL2)
......
gL(aL1xN + bL1) g
L(aL2xN + bL2)
,
where bil is bias and ail = [wi1, wi2, · · · , wiK ] (i =1, 2, · · · , L; l = 1, 2) is input weights, respectively;they are randomly generated. By the definitionof the IT2 fuzzy sets’ lower MF, the relationshipbetween input xt and expected output yt couldbe approximated by the lower MF matrix. Theprincipal MF matrix Φ ∈ R
N×L×1 and the upperMF matrix Φ ∈ R
N×L×1 can also be structuredsimilarly, with the following formulas
Φ:,1,2 =
g1(a11x1 + b11) g1(a12x1 + b12)...
...g1(a11xN + b11) g1(a12xN + b12)
,
...
Φ:,L,2 =
gL(aL1x1 + bL1) gL(aL2x1 + bL2)...
...gL(aL1xN + bL1) gL(aL2xN + bL2)
,
It also forms the upper membership functionvalue-filled tensor, the slices of tensor are
Φ:,1,3 =
g1(a11x1 + b11) g1(a12x1 + b12)...
...g1(a11xN + b11) g1(a12xN + b12)
,
...
Φ:,L,3 =
gL(aL1x1 + bL1) gL(aL2x1 + bL2)...
...gL(aL1xN + bL1) gL(aL2xN + bL2)
.
where aij (i = 1, 2 · · · , L; j = 1, 2) is a randomgenerated weighted vector constructed by tensor.
Remark 1 The uncertain weight method (Runkleret al, 2018) is applied to compute the principal MFmatrices Φ, which reflects the impact of MF valueon the whole defuzzification result of the set. Themapping results are obtained as follows:
uUW (x) =1
2(u(x) + u(x)) · (1 + u(x)− u(x))ζ , (3)
where ζ > 0 measures the uncertainty of lower mem-bership value on type reduction results through theformula (1 + u(x) − u(x))ζ . The uncertainty weightmethod expands the simple method to obtain themean value of upper and lower MF value. The legibleoutput can be given as follows:
(1 + u(x)− u(x))ζ =
0, u(x) = 0, u(x) = 1,1, u(x) = u(x).
(4)
Formula (4) shows that for ζ = 1, the defuzzificationresults increase linearly with uncertainty; the defuzzi-fication results are less than linear when ζ < 1; thedefuzzification results would be greater than linearwhen ζ > 1.
Remark 2 For the expansion of fuzzy sets from type-1 fuzzy MFs to IT2 fuzzy set, when (4) is used fordefuzzification, the extension of IT2 fuzzy set to theenhanced part is called IT2-RVFL (interval type-2random vector function link).
Finally, a 3-tensor Φ ∈ RN×L×3 is established
by the three foregoing membership functionΦ:,:,1,
Φ:,:,2 and Φ:,:,3. Thereinafter, because of the rel-evant usage of A in tensor equations, Φ will bechanged to another capital letter A in the nextsection. It can be known from the relevant contentof the tensor equation that in the enhanced nodeof TT2-RVFL, the weighting matrix is a matrix ofL× 3, so the weighting matrix of TT2-RVFL canbe defined as
X 1 =
wL+1 1 wL+1 2 wL+1 3
wL+2 1 wL+2 2 wL+2 3
......
...w2L 1 w2L 2 w2L 3
. (5)
To TT2-RVFL, the output model can be fused intoone matrix by the following equation
Y1 = αA1X 1 + (1− α)X1Ω1. (6)
Springer Nature 2021 LATEX template
Networks 5
where α ∈ [0, 1] is called the equilibrium coeffi-cient of TT2-RVFL, A1 is denoted as mappingconsequences of non-linear interval type-2 activa-tion function, X 1 is the weighted matrices for theenhanced part, matrix Y1 = [y y]; define ai (i =1, 2 · · · , L) as a weight vector from input layer tothe intensification nodes are stochastically gener-ated, in such a way, the activation functions in Φ,Φ are not fully saturated; X1 is the input matrixthat is structured by input samples, and Ω1 =[ωω] is used to denote the unresolved input weightmatrix.
Remark 3 In the light of equation (6), when α =0, TT2-RVFL will be degenerated to RVFL; when0 < α < 1, TT2-RVFL is a mixed model of tensor-based extreme learning machine and RVFL; whenα = 1, TT2-RVFL will be transformed to tensor-basedextreme learning machine.
Output Layer
Input Layer
Tensor
Tensorconstructions
unfoldings
Fig. 3 Structure of the TT2-ELM network.
2.2 Tensor based type-2 ELM
The TT-2ELM (tensor based type-2 extremelearning machine) was first proposed in (Huanget al, 2019). The advantage of tensor structure isthat the information of quadratic MF can be con-tained and modelled directly into one high dimen-sional array. This characteristic can avoid thetype reduction operation in the process of type-2 fuzzy reasoning. Therefore, the tensor structurecan seamlessly embed type-2 fuzzy sets into ELM.
Fig. 3 intuitively shows the structure of TT2-ELM. Obviously, TT2-ELM is a single hiddenlayer feed forward neural network. For the spec-ified test dataset, which is expressed as (xt, yt),where xt = (xt1, xt2, · · · , xtK) ∈ R
K representsinputs, and yt ∈ R represents outputs. The
mathematical model of TT2-ELM is formed as
A2 ∗N X2 = Y2, (7)
where A2 ∈ RI1···2×J1···2 can be reshaped by Φ
with a specified size N × L × 3, the N × L × 3is regression tensor’s dimension, N is the train-ing patterns, X2 ∈ R
J1···2 is the value of outputweight, Y2 ∈ R
I1···2 is the output matrix, Obvi-ously, when N = 2, equation (7) degenerates intomatrices case.
According to (Huang et al, 2019) (Theorem1), for multi-linear system (7), if there exist anyX2 ∈ R
J1···N , then the multi-linear system (7) issolvable, and the solution of the equation (7) is
X2 = A(1)2 ∗N Y2, (8)
where A(1)2 is a solution of A2 ∗N X2 ∗N A2 = A2.
The resultant X2 in formula (8) is the solution ofTT2-ELM.
For the equation (7), if no X2 ∈ RJ1···N can
be obtained via the analysis, then the multi-linearsystem (7) is unsolvable, and the equation (7) hasa minimum norm solution alternatively.
The following tensor equation can be obtained.
(AT2 ∗N A2) ∗N X2 = AT
2 ∗N Y2. (9)
The gain tensor AT2 ∗N A2 in equation (9) can
be considered as ’square tensor’, and the solutionof tensor equation can still be obtained. Based oftheory of tensor and kernel, the following equationcan be determined.
A2 ∗N (A(1) ∗N A2) ∗N Z = A2 ∗N Z, (10)
where Z is an random tensor that can satisfy for-mula (10). As long as this tensor has a suitableorder, it can be established.
A generally accepted condition is (I −A(1)2 ∗N
A2)∗NZ gratifies A2∗N (I−A(1)2 ∗NA2)∗NZ = 0.
This condition indicates that (I−A(1)2 ∗NA2)∗NZ
is included in the null space of A2 ∗N X2 = 0.Therefore, A general solution can be obtained asfollowing
X2 = A(1)2 ∗N Y2 + (I − A
(1)2 ∗N A2) ∗N Z, (11)
Springer Nature 2021 LATEX template
Networks
where A(1)2 represents the 1-inverse of A2. The
minimum solution for the tensor equation consistsof two parts, one of which comes from the nullspace of equation A2∗NX2 = 0. Note that the ten-sor equation that is formulated by (7) and (11),we have
X2 =(AT2 ∗N A2)
(1) ∗N AT2 ∗N Y2+
(I − (AT2 ∗N A2)
(1) ∗N (AT2 ∗N A2)) ∗N Z.
(12)
For equation (12), if Z = 0, X2 is the mini-mizer. In the light of Corollary 2.14(1) (Beheraand Mishra, 2017), the existence of (AT
2 ∗N A2)(1)
makes
A+2 = (AT
2 ∗N A2)(1) ∗N AT
2 . (13)
Subsequently, the least squares solution ofequation (13) is shown below
X2 = A+2 ∗N Y2. (14)
Equation (14) is the minimum norm solution ofequation (7).
2.3 Tikhonov Regularized OP-ELM
In (Miche et al, 2010), Yoan Miche et al. first pro-posed OP-ELM, it is an improvement of ELM.Fig. 4 shows the Tikhonov regularized ELM(TROP-ELM) (Miche et al, 2011a). First, theOP-ELM network uses three different types ofactivation functions to form the kernel, and theassembled kernel is able to improve robustnessand generality. For the initial condition of theELM algorithm, Sigmoid kernel is used in itsstructure, while OP-ELM could use linear kernel,Sigmoid kernel and Gaussian kernel. Second, com-pared with the original proposed ELM, the multi-response sparse regression algorithm (MRSR) andthe verification method leave one out (LOO)are also introduces in OP-ELM. The main roleis to prune irrelevant variables by pruning therelated neurons of SLFN constructed by ELM.The MRSR algorithm can rank neurons accord-ing to the usefulness of neurons, and the actualpruning technique is performed by leave one outvalidation results.
SLFN Construcation using ELM
Ranking of the best neurons
by LARS: L1 -regularization
Selection of the optimal number of neurons
by TR-PRESS: L2 -regularization
Data
Model
Fig. 4 Implementation steps of TROP-ELM
3 Tensor based hybrid single
fuzzy neural networks design
In the previous section, three single fuzzy neu-ral networks, that are, the TT2-ELM, TT2-RVFLand TROP-ELM are briefed. In this section, thethree networks are stacked into one network, anda regression method that is based on the threeregression results are introduced. The architectureis shown in Fig. 5.
The framework of tensor-based stacked singlefuzzy neural network that is design based on threesingle fuzzy neural networks (TT2-ELM, TT2-RVFL and TROP-ELM), while the three algo-rithms have been proposed by researchers. Thelayer 2 is the hidden layer that is constituted bythe three algorithms to construct a tensor struc-ture. The tensor structure could be constructedin layer 3, then the tensor is unfolded into threedifferent matrices for each single network. Thefinal regression uses a simple normalized scalarweighting, which is the part to be optimized in thefuture.
The unfolding of tensor could use the defini-tions from (Yu et al, 2019), which are appendedas definition 1.
Definition 1 (m,ν)-unfolding. Consider an ten-sor with N -dimensional, and the tensor follows thedimension that A ∈ R
J1×J2×···×JN , χ(m,ν) is a
k-parti-tion of m ∈ AnN , where ν ∈ A
k+1n+1 and
1 = ν1 < ν2 < . . . < νk+1 = n + 1. The (m, ν)-unfolding of A, denoted by ATµ,nuU, is a tensor with
the size of(
∏ν2−1ν=ν1
Jmν
)
× · · · ×
(
∏νk+1−1ν=νk
Jmν
)
×
Jm1 × · · · × JmN−nsuch that the (j1, j2, . . . , jN ) -th
entry of A is the entry of ATm,νU at the position
Springer Nature 2021 LATEX template
Networks 7
TT2-RVFL
TROP-ELM
x1
x2
xn
β1
β2
β3
yo
TT2-ELM
Fig. 5 The structure of TensorSFNN.
( (jm1 ; Jm1) , . . . , (jmk; Jmk
) , jm1 , . . . , jmN−n),
where J = [J1 J2 . . . JN ]T , m ∈ AN (m) and (·; ·)is the linear index of a multi-dimensional array, see,e.g., (Baranyi, 2016) and (Baranyi et al, 2014).
For example, m = [m1 m2 m3]T = [1 2 3]T ∈
R3, ν = [1 3 4]T ∈ A
2+13+1, then χ(m, ν) =
m1 m2 is a 2-partition of m, where m1 =[m1 m2]
T = [1 2]T , m2 = m3 = 3.
Remark 4 By comparing the structures of TT2-RVFL,TT2-ELM and TROP-ELM, we have made a few mod-ifications to TT2-RVFL and TT2-ELM. The originalactivation function of TT2-RVFL and TT2-ELM isSigmoid function, while TROP-ELM uses linear acti-vation function and Gaussian activation function inaddition to Sigmoid function. In order to help ensurethat their structures correspondingly and facilitatethe composition of tensor structures, linear activa-tion function and Gaussian activation function arealso added to TT2-RVFL and TT2-ELM. After addingtwo distinct activation functions, TT2-ELM and TT2-RVFL obtained better performance than the originalproposed ones based on our test.
The 3-tensor A ∈ RN×L×3 generates three
matrices, and the row number of matrices isjust equal to sample number. Three matrices aredenoted by N1, N2 and N3, respectively, and Nk ∈R
N×L, k = 1, 2, 3. The three matrices, N1, N2
and N3 could reconstruct the tensor A ∈ RN×L×3
easily, that is, N1 is the first aspect of A(:, :, 1),which is the mapping results from the TT2-RVFL.N2 is the second aspect of A(:, :, 2), which is themapping results from the TT2-ELM. and N3 isthe three aspect of A(:, :, 3), which is the mappingresults from the TROP-ELM.
To the type-2 fuzzy networks, the LMF, UMFand defuzzification of the secondary member-ship function are used to solve the consequentparameters’ learning problem. To the regression
layer, N1, N2 and N3 are three matrices that isunfolded from the tensor, regression equation canbe denoted as NiWi = yti, where Ni ∈ R
N×L×3,Wi ∈ R
L×3 and yt ∈ RN×1.
To make the network perform best, first, theerror of each network is calculated, in addition,root mean square error is used as the measurementstandard. Second, the best-performing networkare found out, which is used to train the other twonetwork errors. After processing, the network isrecorded as y′t. Finality, The output are averagedbased on y′t, that is
ykt =
3∑
k=1
αky′
t,
3∑
k=1
αk = 1 (15)
which is the average results of the three type-2fuzzy networks, the MF values are obtained fromlower MF and upper MF, since the matrices arethe unfolded results from the tensor.
And the defuzzificaiton result could be calcu-lated from secondary MFs, it could also be usedfor the stacked SFNN. The unconstraint regres-sion result of Akβk = yt could be denoted asβk = (AT
kAk)−1AT
k yt. In statistics, this methodis known as ridge regression, it is associated tothe Levenberg–Marquardt algorithm and AndreyTikhonov method to solve the regularization ofill-posed non-linear least-squares problems. Sup-pose that for a known matrix Nk and vector yt, avector x is expected to be found, such that
Akβk = yt. (16)
In most of the cases, ordinary least squaresestimation leads to an overdetermined (over-fitted), or more often an underdetermined (under-fitted) system of equations. Therefore, in solvingthe ill-posed inverse-problem, the inverse mappingoperators that has the undesirable tendency of
Springer Nature 2021 LATEX template
Networks
amplifying noise (The eigenvalue value is max-imum in the reverse mapping and the singularvalue is minimum in the forward mapping). More-over, every element of the reconstructed version ofx is implicitly nullified by ordinary least squares,instead of taking a model to as a prior of x. For thepurpose of the minimize residuals sum of squares,and the particular solution also satisfies some suit-able qualities, a regularization term which can beadded to this primary minimization problem, itcan be succinctly scripted as follows
‖Akβk − yt‖2 + ‖Γkβk‖
2, (17)
where ‖·‖ is the Euclidean norm, Γk is a appropri-ately selected Tikhonov weighting matrix.
Under many circumstances, the matrix Γk isselected as a multiple of the character matrix αkI,By L2 regularization, solutions with smaller normscan be found (Ng, 2004). At other times, in theevent that the fundamental vectors are consid-ered primarily uninterrupted, a low-pass operatorcan be accustomed to strengthen flatness. Thiscanonicalization enhances the conditioning of theproblem, which leads to a straightforward numeri-cal solution. An approximated solution is signifiedthrough x, which is presented by:
βk = (ATkAk + ΓTΓ)−1AT
k yt. (18)
the individual algorithm of the stacked singlefuzzy neural network could use the regularizedresults for the tensor unfolded structure’s learningmethod.
4 Simulation results for the
datasets
In this section, the UCI benchmark dataset andthe other four actual datasets are tested to evalu-ate the performance of this method. Among all thesimulations, root mean square error (RMSE) wasutilized to assess the performance of the TSFNNin this paper and the four comparison methods,which is given by
RMSE =
√
√
√
√
1
N
N∑
t=1
(yt − yt)2 (19)
where yt is the predicted signal, yt is the targetsignal, and N is the length of the testing sequence.
All experiments are performed on a computerwith AMD Ryzen 7 4800U with Radeon Graphics1.80 GHz and 16 GB RAM. The result of total5000 times were performed on each data set.
the Abalone, Airfoil self noise, Auto-Mpg, Bank, Concreteslump, Diabetes, Delta aileron, Delta elevators, Energy eff-iciency and Wine quality white datasets could be downloadvia the following url:https://archive.ics.uci.edu/ml/datasets.php
4.1 Regression problems
In this section, ten realistic world regression prob-lems are used for testing. Abalone is a datasetthat is used to predict abalone’s age by physicalmeasurements, which includes the whole weight,shucked weight and viscera weight of abalone inTasmania, and 4447 samples with 9 attributes isincluded in the dataset. Airfoil Self-Noise datasetis obtained from a series of aerodynamic andacoustic tests on two-dimensional, and three-dimensional airfoil blade profiles in an anechoicwind tunnel, it contains 1503 samples and 6attributes. The data set of Auto-MPG assemblesmiles each gallon data with dissimilar car brands,it contains 392 samples with 8 attributes.
The bank dataset simulates the customer’spatience who select their favored services in thebank according to 8 factors, for example residen-tial area, distance, virtual temperature regulatingbank option and so on, it contains 8192 sampleswith 8 attributes. Concrete Slump dataset con-tains information about the factors that affectslump flow of concrete, it includes 103 sampleswith 11 attributes. Diabetes is a dataset that
investigates the reliance of the grade of serum C-peptide on various factors, it can be used to mea-sure residual insulin secretion patterns, 768 sam-ples with 4 attributes are included in the dataset.Delta ailerons and Delta elevators are recordedailerons’ data and elevators’ data for delta, andthey have 7129 samples with 6 attributes, and9247 with 7 attributes, respectively. Energy effi-ciency is a dataset that is obtained by energyanalysis of 12 various architectural shapes, whichis simulated in Ecotect, and there are 768 sam-ples and 8 features in it, the regression problemis for forecast 2 authentic valued responses thatare cooling load and heating load. Wine qualitywhite is a dataset associated with red and whitewine samples, it contains 4898 samples with 12attributes.
The information of the dataset is presentedin Table 1, these datasets involve four small-scale datasets and six moderate-scale datasets.The mean and standard of 5000 experimentalresults of Abalone, Airfoil self-noise, Auto-Mpg,Bank, Concrete slump, Diabetes, Delta ailerons,Delta elevators, Energy efficiency and Wine qual-ity white are showed in Table 2. TT2-RVFL,TT2-ELM, OP-ELM and TROP-ELM are usedfor algorithm comparison.
Results of Friedman test on these ten datasets for five method (TT2-RVFL, TT2-ELM, OP-ELM, TROP-ELM and TSFNN) are listed inTable 3. It can be inferred that the training errorand testing error of the proposed method is thesmallest. The stacked tensor-based hybrid sin-gle fuzzy neural networks of proposed can shareadvantages of TT2-RVFL, TT2-ELM and TROP-ELM, and the tensor quantization method of fuzzysystem may be a method of extending type-2 fuzzymodeling.
Table 3 Results of Friedman test on ten datasets (Thetesting results are listed in bracket).
In this section, four datasets which are ElectricalFault detection and classification datasets, Aster-oid Dataset, Novel Corona Virus 2019 Dataset andWind Power Generation Data are used for testingmodel performance.
Power systems consist of many complex dynamicand interactive elements that are always vulner-able to interference or electrical failures. Trans-mission lines are the most critical part of thepower system, and the prominent role of trans-mission lines is to transmit electricity from thesource area to the distribution destination in thenetwork. The faults of power system transmissionlines should be first correctly detected and clas-sified, and should be eliminated in the shortestpossible time. Electrical Fault detection and clas-sification dataset contains the current and voltageof the line under different fault conditions (Jamilet al, 2015). The dataset contains the detection ofpower system faults and classifying fault types forthe power system faults. The dataset of power sys-tem fault detection contains 12001 sampling datawith six inputs (Ia, Ib, Ic, Va, Vb, V, c) and twooutputs (0 and 1), no fault is denoted by 0, andfault is denoted by 1.
The dataset of classifying fault types contains7861 group data with six inputs (Ia, Ib, Ic, Va,Vb, V, c) and four outputs (G, C, B and A), theyrepresents 4 generators of 11× 103V, respectively.No fault occurs is denoted by 0, and fault occurs isdenoted by 1. The combination of G, C, B and Arepresents various failures, and the failures faultsare shown in Table 5.
The faults of the system are judged accordingto the current and voltage of the power system.The dataset consists of two outputs that repre-sents whether the system faults. Fig. 6 shows thatno faults in Power Electrical detection dataset.Similarly, Fig. 7 shows that faults occurs in PowerElectrical detection dataset. The transverse axisdelegates samples, and the longitudinal axis del-egates the values of Va, Vb, Vc, Ia, Ib and Ic.Compared with Fig. 6 and Fig. 7, if there is nofault in the electric power system, the values ofcurrent and voltage are generally stable, and itschange trend is similar to the sin function, which
is also consistent with the characteristics of ACin the electric power system. However, once afault occurs in the power system, the values ofcurrent and voltage are abnormal, which repre-sents different fault locations. This anomaly canbe clearly seen from Fig. 7. The comparison resultof detection of power system faults dataset that isobserved to determine whether the power systemis faulty is shown in Table 4.
Fig. 6 The data of no faults in Electrical detect dataset.
Fig. 7 The data has faults in Electrical detect dataset.
Comparison results of TT2-RVFL, TT2-ELM,OP-ELM, TROP-ELM and TSFNN on the Elec-trical Fault detection dataset are used to testthe performance of the five algorithms. Results ofTSFNN in the Table 4 show that the generaliza-tion ability of TSFNN is better than the other fouralgorithms. Moreover, data indicats power systemfailure can be considered as no power system fail-ure added noise by comparing with Fig. 6 and Fig.
Springer Nature 2021 LATEX template
Networks 11
7. Therefore, Table 4 also show that the distur-bance reject ability of TSFNN is better than theother four algorithms.
Table 4 Comparison results with TSFNN, TT2-RVFL,TT2-ELM, OP-ELM and TROP-ELM for Electrical-class,Electrical-detect (A brief introduction is listed in Table 1).
For Electrical Fault classification case, wedecompose Electrical Fault classification datasetinto five parts according to the different locationsof faults. It can be seen from Table 5 that thedataset provides six kinds of faults, but we do notfind LL fault ([G, C, B, A] = [0, 0, 1, 1]) in thedataset, which represents the fault between PhaseA and Phase B.
Through the above analysis, we know thatin the power system, the faults data can beregarded as the no faults data with added noise.Thus, in the five extracted datasets, the datasetof LG fault, LLG fault, LLL fault, and LLLGfault can be treated as imposing different noiseon the dataset with no fault status. Moreover, theextracted dataset represents only one power sys-tem fault, so its data is more pure and has moreobvious characteristics and trends. Through theabove-mentioned analysis, the anti-interferenceperformance and generalization performance ofthe algorithm can be further verified. Resultsof TSFNN in the Table 6 fully proof that theexcellent disturbance rejection and generalizationperformance of TSFNN.
Table 5 Faults represented by G, C, B and A.
[G, C, B, A] Faults[0, 0, 0, 0] No fault[1, 0, 0, 1] LG fault (Between phase A and phase G)[0, 0, 1, 1] LL fault (Between phase A and phase B)[1, 0, 1, 1] LLG fault (Between Phases A, B and ground)[0, 1, 1, 1] LLL fault (Between all three phases)[1, 1, 1, 1] LLLG fault (Three phase symmetrical fault)
4.2.2 Asteroid Dataset
The Asteroid Dataset is officially maintained byJet Propulsion Laboratory of California Instituteof Technology, which is an organization under
Table 6 Comparison results with TSFNN, TT2-RVFL,TT2-ELM, OP-ELM and TROP-ELM for various faultsin Electrical dataset. (A brief introduction is listed inTable 1).
NASA. The data set is publicly available in JPLSmall-Body Database Search Engine. This datasetcould also be obtained from kaggle. Table 7 showsbasic column definition for Asteroid dataset. Aportion of the data is extract by us as a compari-son test dataset when using Asteroid dataset. The2500 samples in the dataset presents 7 attributes,these samples are applied to validate the pro-posed algorithm. These properties are Geometricalbedo, Eccentricity, Semi-major axis, inclinationangle about the x-y elliptic plane, Earth Mini-mum Orbit Intersection Distance and RMS for theAsteroid, respectively.
The comparison results with five methods aredemonstrated in Table 8. The results demonstratethat the TSFNN performs best with respect totraining error, while TT2-RVFL and TT2-ELMperforms best in testing error. Meanwhile, the per-formance of OP-ELM and TROP-ELM is bad.Because the approach of proposed in this paperis superposition of TT2-RVFL, TT2-ELM andTROP-ELM. The main reason why TSFNN, OP-ELM and TROP-ELM perform well in trainingand poor in testing is that these three methods alluse multi-response sparse regression (MRSR), it isa variable sorting technology that is extended from
Object ID Object internal database IDObject fullname Object full name/designation
pdes Object primary designationname Object IAU nameNEO Near-Earth Object (NEO) flagPHA Potentially Hazardous Asteroid (PHA) flagH Absolute magnitude parameter
Diameter object diameter (from equivalent sphere) km UnitAlbedo Geometric albedo
Diameter sigma 1-sigma uncertainty in object diameter km UnitOrbit id Orbit solution IDEpoch Epoch of osculation in modified Julian day formEquinox Equinox of reference frame
e Eccentricitya Semi-major axis au Unitq perihelion distance au Uniti inclination; angle with respect to x-y ecliptic planetp Time of perihelion passage TDB Unit
moid ld Earth Minimum Orbit Intersection Distance au Unit
the least angle regression algorithm (Simila andTikka, 2005) and (Efron et al, 2004). Accordingto the usefulness of neurons, the MRSR algorithmcan obtain a ranking of the neurons in OP-ELM(Miche et al, 2010). TROP-ELM is an improve-ment of OP-ELM, the MRSR method is also usedfor the input data of TROP-ELM. Due to theproposed TSFNN includes TROP-ELM, thus theTSFNN is affected by MSRS method.
To the MRSR, an important feature is thatthe obtained ordering is exact in the case of lin-ear problems. The Asteroid dataset collects theattributes of asteroid. A part of its data is used,it is also nonlinear. And the OP-ELM and theTROP-ELM have one detail that the neural net-works they constructed are linear between thehidden layer and the output layer, the role ofMRSR algorithm is that will get an exact rankingof the neurons. The sequence obtained by sortingcan be used to sort the kernels of model. Whenthe whole dataset is nonliner, so the exact rankingof neurons cannot be obtained by OP-ELM. Sim-ilarly, TROP-ELM and TSFNN are also affectedby this flaw. Therefore, TSFNN performs well inthe training part, and in the test part, due toMRSR method, the extracted data set featurescannot be well applied to the testing set, result-ing in poor performance of TSFNN in the testingphase.
According to the data in Table 8, the per-formance of TT2-RVFL and TT2-ELM is thebest. The TT2-RVFL and the TT2-ELM are con-structed by tensor structure and interval type-2fuzzy sets. The membership degree of type-2 fuzzyset is characterized by type-1 fuzzy set. Since the
Table 8 Comparison results with TSFNN, TT2-RVFL,TT2-ELM, OP-ELM and TROP-ELM for Asteroid (Abrief introduction is listed in Table 1).
type-1 fuzzy set has a strong ability to deal withuncertainty in the system, so type-2 fuzzy setgreatly strengthens the processing ability of fuzzysystem for uncertainty and nonlinearity, and ithas good performance in nonlinear systems withhigh uncertainty. Therefore, type-2 fuzzy systemshave strong generalization ability. And the tensorstructure is also good at dealing with uncertainsystems, which can improve the generalizationperformance of the system.
The merits of type-2 fuzzy sets, and the ten-sor structure are inherited by the tensor-basedtype-2 fuzzy system. On the basis of the aboveanalysis, TT2-RVFL and TT2-ELM have goodperformance on Asteroid dataset, their trainingerror and testing error are minimal in Table 8.
From the test performance of TSFNN, sinceTSFNN contains TT2-RVFL and TT2-ELM, itmakes up for the defect of insufficient generaliza-tion ability of TROP-ELM in nonlinear system.This also proves the excellent generalization abil-ity of type-2 fuzzy systems, the advantages of thestacked tensor-based hybrid single fuzzy neuralnetworks indicate that the stacked way of net-works designing can inherit the merits of the usedalgorithms, and the stacked structure of the threealgorithms are complementary with each other.
4.2.3 Novel Corona Virus 2019 Dataset
COVID-19 affected cases in the Corona Virus2019 dataset contains date information label. Thisdataset contains daily level information about thenumber of affected cases, the number of deathsand the rehabilitation of the new coronavirus in2019. It is worth noting that this is a time seriesdata, so any number of cases given a fixed date iscumulative. the data from national centers for dis-ease control and prevention are collected in github.The data is updated daily. Eight cities in Beijing,Shanghai, Chongqing Tianjin, Arizona, Washing-ton, California and Illinois are used for testing,
and the time stamp ranges from 22 Jan, 2020 to29 May, 2021.
There is no doubt that the extracted data ofeight cities is composed of time series dataset,and is a small-scale dataset with three attributes.Four of the eight cities that are selected were fromChina, four were from the United States, the out-breaks in both regions were predicted. Althoughthe data set itself is a small-scale one, because theattribute of the data set is only three, obviously,attribute is not sufficient. The performance of theproposed network is tested in the case that thefeature attributes of the dataset are insufficient.
Table 9 Comparison results with TSFNN, TT2-RVFL,TT2-ELM, OP-ELM and TROP-ELM for Covid19 Beij-ing, Covid19 Shanghai, Covid19 Tianjin, Covid19 Chon-gqing, Covid19 Arizona, Covid19 Washington, Covid19California, Covid19 Illinois (A brief introduction is listedin Table 1).
The comparison results are shown in Table9. Fig. 8 is the results that is carried out basedon samples and attributes from four datasets,Beijing, Shanghai, Tianjin and Chongqing. Fromthe results, TSFNN has the best performancecompared with the other four algorithms. Bycomparing the data results in Table 9, on thewhole, the data results of the five comparisonmethods are similar. Moreover, from the dataresults, the results of Beijing, Shanghai, Tianjinand Chongqing are significantly worse than thoseof Arizona, Washington, California and Illinois.
Similarly, the same operation is performed onthe four datasets of Arizona, Washington, Califor-nia and Illinois, and the results are shown in Fig.9.
(a) Beijing (b) Shanghai
(c) Tianjin (d) Chongqing
Fig. 8 The data for Novel Corona Virus 2019 Dataset inBeijing, Shanghai, Tianjin and Chongqing.
It can be regarded from Fig. 8 and Fig. 9that, due to the characteristics of the COVID-19 dataset itself, the overall trend of the datais rising, reflecting the characteristics of its timeseries, and the growth rate of the curve reflectsthe situation of the new coronavirus. From thefigures, these eight datasets are more suitable forregression problems, and the data is relatively sta-ble. Therefore, there is little difference betweenTSFNN and four comparison methods in Table 9.
Compared with Fig. 8 and Fig. 9, the curve inFig. 9 is smoother and the overall trend is moreobvious. Although the curve in Fig. 8 shows anoverall upward trend, the data soon go back to
Springer Nature 2021 LATEX template
Networks
(a) Arizona (b) Washington
(c) California (d) Illinois
Fig. 9 The data for Novel Corona Virus 2019 Dataset inArizona, Washington, California and Illinois.
the stable state. Therefore, the data in Fig. 9 ismore suitable for forecasting regression problemsthan the data in Fig. 8, which is also the reasonthat in Table 9 the overall performance of the datasets corresponding to four cities in China on fivealgorithms is not as good as that of the data setscorresponding to four cities in the United States.
Through the above analysis, the data char-acteristics of the data set itself lead to datadifferences in table 9. Because of this difference,four data sets in China are comparable to fourdata sets in the United States. On the premise thatthe eight data sets are small-scale time series, thedata characteristics of the four data sets in Chinaare quite different, and the performance in theregression problem is poor, while the data in theUnited States are more obvious and more suitablefor regression analysis.
In addition, four data sets such as Beijing canbe regarded as data sets with more complex datastructures than four data sets such as Arizona,and the features are more diverse, not limited toreflect the upward trend of data. Obviously, com-bined with the actual current situation. The fourdata sets related to the United States can betterpredict the future epidemic situation in the UnitedStates. As far as the data in the data set itself isconcerned, the four data sets such as Arizona aremore ”pure” and suitable for regression problems.Therefore, the performance of the five methods on
these four data sets is better, while the perfor-mance of Beijing and other four and data sets isrelatively poor.
According to the overall data of Table 9,TSFNN still performs best compared with theother four comparison methods. In four datasetsof Beijing, Shanghai, Tianjin and Chongqing,although the overall data is relatively poor due tothe problems with the data itself, the performanceof TSFNN is still the best in these four datasets.TSFNN also performs best in Arizona, Washing-ton, California and Illinois datasets which has thebetter data than 4 datasets for China. This showsin the excellent feature extraction and generaliza-tion ability of TSFNN. It is demonstrated that thescheme of tensor stacked neural network can inte-grate the advantages of its members and enhancethe ability of data feature extraction.
4.2.4 Wind Power Generation Dataset
The wind power generation or wind energy is theuse of wind power to provide mechanical powerthrough wind turbines and generate electricityby rotating generators. Wind power is a popularand sustainable renewable energy, and its impacton the environment is much smaller than burn-ing fossil fuels. Wind power generation dataset iscollected from four German transmission systemoperators which are 50Hertz, Amprion, TenneTTSO and TransnetBW, the date ranges from 23August 2019 to 22 September 2020. It containsnon-normalized power generation data with 15minutes per interval, a total of 96 sets of datawith 96 points a day. The measurement unit ofthese data is terawatts hours. Obviously, there are397 samples with 96 attributes in Wind powergeneration dataset.
The Wind Power Generation dataset is a smalland medium-sized dataset. Through the data set,the prediction ability of the model can be veri-fied in the case of sufficient data set attributes.Meanwhile, 50Hertz, Amprion, TransnetBW andTenneT TSO manage Germany’s east, south, westand north regions, respectively; and the same istrue of their geographical location. For wind powersystems, power generation in different geographi-cal locations has different characteristics, and thisdata difference caused by different geographicallocations is also affected by the seasons. In Ger-many, the Officially spring is during March, April
Springer Nature 2021 LATEX template
Networks 15
and May, the Summer is taken from June throughto August. The autumn is during the months ofSeptember, October and November and winter isfrom December to February. Obviously, geograph-ical and seasonal factors are important for windpower systems. Therefore, among the 96 attributesof this data set, by virtue of the characteristicsof the wind power generation system, the entiredata set will have different characteristics in dif-ferent months. Thus, these 96 attributes containrich feature information, which can better assessthe performance of the model.
Table 10 Comparison results with TSFNN, TT2-RVFL,TT2-ELM, OP-ELM and TROP-ELM for Hertz50, Ampr-ion, TenneTTSO, TransnetBW (A brief introduction is lis-ted in Table 1).
The comparison results of TSFNN and TT2-RVFL, TT2-ELM, OP-ELM and TROP-ELM on50Hertz, Amprion, TenneT TSO and TransnetBWdatasets are shown in Table 10. Overall, resultsin the Table 10 show that the training error ofTSFNN on four datasets is the best, while thetest error is the best on the data sets of Hertz50,Amprion and TenneT TSO, and the worse onTransnetBW dataset. For Hertz50, Amprion andTenneT TSO datasets, TSFNN, OPELM andTROP-ELM perform better than TT2-RVFL andTT2- ELM, especially for training error and test-ing error, TT2-RVFL and TT2-ELM are muchworse than the other three algorithms. Among thethree methods of TSFNN, OP-ELM and TROP-ELM, TSFNN has better performance.
This result shows the excellent generalizationability and feature extraction ability of TSFNN.In the superposition of TSFNN, its members TT2-RVFL and TT2-ELM are also improved, as shownin Remark 4. The comparison results of TSFNNwith TT2-RVFL and TT2-ELM prove that thismethod of composing kernel space with multipleactivation functions can enhance the performanceof the network. For TransnetBW dataset, theresults in Table 10 show that TSFNN, OP-ELMand TROP-ELM perform better on the trainingset than TT2-RVFL and TT2-ELM.
On the testing set, TT2-RVFL and TT2-ELMperform better than the other three methods.Moreover, the performance of TSFNN, OP-ELMand TROP-ELM on testing set is far from that ontraining set. And the performance of TT2-RVFLand TT2-ELM is normal, and there is no differ-ence significantly between the performance on thetesting set and the training set. This shows thatthe generalization ability of type-2 fuzzy systemis better than the ability of type-1 system, due tothe TROP-ELM is a member of TSFNN network,the overall test performance of TSFNN is poor.Because the performance of TSFNN on the train-ing set shows that it has good feature extractionability on the data set, better results are obtained,and the results of OP-ELM and TROP-ELM arealso this reason. At the same time, OP-ELM andTROP-ELM also benefit from the MRSR func-tion. The MRSR function pairs are sorted, andthen through the pruning links in OP-ELM andTROP-ELM, better results will be obtained forthe data set with more attributes. This sort andpruning result on the training set may not be suit-able for test set data, especially when the WindPower Generation dataset has 96 attributes. Thisshows that the sorting and pruning of the networkhas a certain limit, and the effect on some datasets needs to be tested, which may produce largesingular values, thus affecting the final results.
In the previous analysis, the stacked tensor-based hybrid single fuzzy neural networks systemproposed in this paper has the advantages of con-centrating its member networks. At the same time,it inherits the defects of its member network. Theperformance of TSFNN and other four compari-son methods on TransnetBW dataset in Table 10proves this phenomenon again. At the same time,it also shows that this ability to concentrate theadvantages of its member networks also has the
Springer Nature 2021 LATEX template
Networks
upper limit. When a member network performspoorly, the advantages of other member networksare difficult to fully supplement this defect.
TSFNN can obtain the advantages of thesemember networks in different problems by com-bining different member networks, and then theseadvantages will be concentrated by tensor super-position. Meanwhle, the defects of some membernetworks are complemented by tensor overlay net-works. However, when the performance of somemember functions is poor to a certain extent,this complementary mechanism is difficult to com-pletely eliminate the defects of the member func-tion. When it exceeds the ”threshold” of thiscomplementary mechanism, the poor performancemember network will dominate the performance ofthe whole network, resulting in poor performanceof the whole network.
5 Conclusions
In this paper, a stacked tensor-based hybrid sin-gle fuzzy neural networks (TSFNN) was proposed,which is a neural network combination model.The TT2-RVFL, TT2-ELM and TROP-ELM areused to form the TSFNN network, and TT2-RVFLand TT2-ELM are optimized by using the ker-nel space method to enhance their performance.TSFNN overlays the hidden layer output networkof its member network by tensor. In this process,the tensor-based superposition system inherits theadvantages of the type-2 fuzzy sets and the fuzzyreasoning ability of the fuzzy system. Simultane-ously, the good performance of the linear systemthat is generated by MRSR method and pruningmethod in TROP-ELM is also extracted by tensorstructure.
Because tensor results can concentrate theadvantages of member networks, TSFNN hasgood generalization ability and anti-noise ability.TSFNN will also inherit the defects of its mem-ber networks. In general, the ability of TSFNN toconcentrate the advantages of member networkswill make up for the shortcomings of some mem-ber networks, but this complementarity is limited,and when the data is too complex, there will stillbe underfitting, par exemple, TransnetBW datasetin Table 10.
By and large, the proposed TSFNN algorithmhas excellent generalization ability, anti-noise abil-ity and feature extraction ability. These capabil-ities are demonstrated and validated on 10 UCIstandard datasets and 4 real world datasets. TheTSFNN algorithm supplements the tensor-basedmodel optimization and model combination meth-ods, indicating that the tensor structure superpo-sition neural network is a feasible neural networkcombination method. For the purpose of obtain afast model of the data set, unfolding of tensor isused, and then the regression results are obtainedby matrix regression. Tensor regression and tensorequation can also be used in this direction, whichis a future optimization direction.
Acknowledgment
This document is the results of the researchproject funded by the National Natural ScienceFoundation of China under number 12161065,61603126 and 61966026, Natural Science Foun-dation of Inner Mongolia (2020MS06016,2019MS01005), and the work is also supportedby the Research Foundation for Advanced Tal-ents Research Foundation of Inner MongoliaUniversity (21700-5185130).
Declaration of conflicting
interest
The author(s) declared no potential conflicts ofinterest with respect to the research, authorship,and/or publication of this article.
References
Akusok A, Veganzones D, Miche Y, et al(2015) MD-ELM: Originally mislabeled samplesdetection using OP-ELM model. Neurocom-puting 159:242–250. https://doi.org/10.1016/j.neucom.2015.01.055
Alencar AS, Rocha Neto AR, Gomes JPP (2016)A new pruning method for extreme learn-ing machines via genetic algorithms. AppliedSoft Computing 44:101–107. https://doi.org/10.1016/j.asoc.2016.03.019
Baranyi P (2016) TP-Model Transformation-Based-Control Design Frameworks. Springer
International Publishing, Switzerland, https://doi.org/10.1007/978-3-319-19605-3
Baranyi P, Yam Y, Varlaki P (2014) Tensor Prod-uct Model Transformation in Polytopic Model-Based Control. CRC Press-Taylor and FrancesGroup, https://doi.org/10.1201/9781315218045
Behera R, Mishra D (2017) Further results ongeneralized inverses of tensors via the ein-stein product. Linear and Multilinear Alge-bra 65(8):1662–1682. https://doi.org/10.1080/03081087.2016.1253662
Bugnon L, Calvo RA, Milone D (2020) Dimen-sional affect recognition from HRV: Anapproach based on supervised SOM and ELM.IEEE Transactions on Affective Computing11(1):32–44. https://doi.org/10.1109/TAFFC.2017.2763943
Christou V, Tsipouras MG, Giannakeas N,et al (2019) Hybrid extreme learning machineapproach for heterogeneous neural networks.Neurocomputing 361:137–150. https://doi.org/10.1016/j.neucom.2019.04.092
Efron B, Hastie T, Johnstone I, et al (2004)Least angle regression. The Annals of Statis-tics 32:407–499. https://doi.org/10.1214/009053604000000067
Grigorievskiy A, Miche Y, Ventela A, et al (2014)Long-term time series prediction using OP-ELM. Neural Networks 51:50–56. https://doi.org/10.1016/j.neunet.2013.12.002
Grigorievskiy A, Miche Y, Kapyla M, et al (2016)Singular value decomposition update and itsapplication to (Inc)-OP-ELM. Neurocomputing174:99–108. https://doi.org/10.1016/j.neucom.2015.03.107
Hashmi AS, Ahmad T (2019) GP-ELM-RNN:Garson-pruned extreme learning machine basedreplicator neural network for anomaly detec-tion. Journal of King Saud University - Com-puter and Information Sciences https://doi.org/10.1016/j.jksuci.2019.09.007
Huang SRN, Zhao GL, Chen MH (2019) Ten-sor extreme learning design via generalizedMoore–Penrose inverse and triangular type-2 fuzzy sets. Neural Computing and Appli-cations 31(9):5641–5651. https://doi.org/10.1007/s00521-018-3385-5
Jamil M, Sharma SK, Singh R (2015) Faultdetection and classification in electrical powertransmission system using artificial neural net-work. SpringerPlus 4(1):334. https://doi.org/10.1186/s40064-015-1080-x
Lai XP, Cao JW, Huang XF, et al (2020)A maximally split and relaxed ADMM forregularized extreme learning machines. IEEETransactions on Neural Networks and Learn-ing Systems 31(6):1899–1913. https://doi.org/10.1109/TNNLS.2019.2927385
Miche Y, Sorjamaa A, Bas P, et al (2010)OP-ELM: Optimally pruned extreme learningmachine. IEEE Transactions on Neural Net-works 21(1):158–162. https://doi.org/10.1109/TNN.2009.2036259
Miche Y, van Heeswijk M, Bas P, et al (2011a)TROP-ELM: A double-regularized ELM usingLARS and Tikhonov regularization. Neuro-computing 74(16):2413–2421. https://doi.org/10.1016/j.neucom.2010.12.042
Miche Y, van Heeswijk M, Bas P, et al (2011b)TROP-ELM: A double-regularized ELM usingLARS and Tikhonov regularization. Neuro-computing 74(16):2413–2421. https://doi.org/10.1016/j.neucom.2010.12.042
Ming YW, Zhu E, Wang M, et al (2018) DMP-ELMs: Data and model parallel extreme learn-ing machines for large-scale learning tasks.Neurocomputing 320:85–97. https://doi.org/10.1016/j.neucom.2018.08.062
Ng AY (2004) Feature selection, L 1 vs. L 2 reg-ularization, and rotational invariance. https://doi.org/10.1145/1015330.1015435
Peng Y, Li QX, Kong WZ, et al (2020) A jointoptimization framework to semi-supervisedRVFL and ELM networks for efficient data clas-sification. Applied Soft Computing 97:106,756.https://doi.org/10.1016/j.asoc.2020.106756
Ragusa E, Gastaldo P, Zunino R, et al (2020)Balancing computational complexity and gen-eralization ability: A novel design for ELM.Neurocomputing 401:405–417. https://doi.org/10.1016/j.neucom.2020.03.046
Runkler TA, Coupland S, John R, et al (2018)Interval type–2 defuzzification using uncer-tainty weights. In: Frontiers in ComputationalIntelligence. Springer, p 47–59, https://doi.org/10.1007/978-3-319-67789-7$ $4
Sekh AA, Dogra DP, Kar S, et al (2020) ELM-HTM guided bio-inspired unsupervised learningfor anomalous trajectory classification. Cogni-tive Systems Research 63:30–41. https://doi.org/10.1016/j.cogsys.2020.04.003
Shi J, Xue ZY, Dai YK, et al (2019) Cas-caded multi-column RVFL+ classifier for single-modal neuroimaging-based diagnosis of parkin-son’s disease. IEEE Transactions on BiomedicalEngineering 66(8):2362–2371. https://doi.org/10.1109/TBME.2018.2889398
Simila T, Tikka J (2005) Multiresponse sparseregression with application to multidimensionalscaling. Springer Berlin Heidelberg, Berlin, Hei-delberg, pp 97–102, https://doi.org/10.1007/11550907 163
Wang PP, Zheng XQ, Ku JH, et al (2020)Multiple-Instance learning approach viabayesian extreme learning machine. IEEEAccess 8:62,458–62,470. https://doi.org/10.1109/ACCESS.2020.2984271
Wei CY, Li CS, Feng C, et al (2020) A T–S fuzzymodel identification approach based on evolv-ing MIT2-FCRM and WOS-ELM algorithm.Engineering Applications of Artificial Intel-ligence 92:103–653. https://doi.org/10.1016/j.engappai.2020.103653
Wong SY, Yap KS, Li X (2020) A new probabilis-tic output constrained optimization extreme
Xu XZ, Shan D, Li S, et al (2019) Multi-labellearning method based on ML-RBF and lapla-cian ELM. Neurocomputing 331:213–219. https://doi.org/10.1016/j.neucom.2018.11.018
Xu XZ, Li S, Liang TM, et al (2020) Sampleselection-based hierarchical extreme learningmachine. Neurocomputing 377:95–102. https://doi.org/10.1016/j.neucom.2019.10.013
Xue ZY, Shi J, Dai YK, et al (2018) Transcranialsonography based diagnosis of parkinson’s dis-ease via cascaded kernel RVFL+. In: 2018 40thAnnual International Conference of the IEEEEngineering in Medicine and Biology Society(EMBC), pp 574–577, https://doi.org/10.1109/EMBC.2018.8512384
Yang J, Cao JW, Wang TL, et al (2020a) Regular-ized correntropy criterion based semi-supervisedELM. Neural Networks 122:117–129. https://doi.org/10.1016/j.neunet.2019.09.030
Yang J, Cao JW, Xue AK (2020b) Robustmaximum mixture correntropy criterion-basedsemi-supervised ELM with variable center.IEEE Transactions on Circuits and Systems II:Express Briefs 67(12):3572–3576. https://doi.org/10.1109/TCSII.2020.2995419
Yang Z, Al-Dahidi S, Baraldi P, et al (2020c) Anovel concept drift detection method for incre-mental learning in nonstationary environments.IEEE Transactions on Neural Networks andLearning Systems 31(1):309–320. https://doi.org/10.1109/TNNLS.2019.2900956
Yaw O, Yap KS, Wong SY, et al (2020) Enhance-ment of neural network based multi agent sys-tem for classification and regression in energysystem. IEEE Access 8:163,026–163,043. https://doi.org/10.1109/ACCESS.2020.3012983
Yildirim H, Revan Ozkale M (2019) The perfor-mance of ELM based ridge regression via theregularization parameters. Expert Systems withApplications 134:225–233. https://doi.org/10.1016/j.eswa.2019.05.039
Yu Q, Miche Y, Eirola E, et al (2013) Regular-ized extreme learning machine for regressionwith missing data. Neurocomputing 102:45–51.https://doi.org/10.1016/j.neucom.2012.02.040
Yu Y, Li Z, Liu XD, et al (2019) A nested tensorproduct model transformation. IEEE Transac-tions on Fuzzy Systems 27(1):1–15. https://doi.org/10.1109/TFUZZ.2018.2851575
Zeng Y, Li Y, Chen J, et al (2020a) ELMembedded discriminative dictionary learning forimage classification. Neural Networks 123:331–342. https://doi.org/10.1016/j.neunet.2019.11.015
Zeng YJ, Chen JC, Li Y, et al (2020b) Cluster-ing via adaptive and locality-constrained graphlearning and unsupervised ELM. Neurocom-puting 401:224–235. https://doi.org/10.1016/j.neucom.2020.03.045
Zhang L, P.N.Suganthan (2016) A comprehensiveevaluation of random vector functional link net-works. Information Sciences 367-368:1094–1105.https://doi.org/10.1016/j.ins.2015.09.025
ridge regression and random vector functionallink ensembles [research frontier]. IEEE Com-putational Intelligence Magazine 12(4):61–72.https://doi.org/10.1109/MCI.2017.2742867
Zhang PB (2020) A new learning paradigmfor random vector functional-link network:RVFL+. Neural Networks 122:94–105. https://doi.org/10.1016/j.neunet.2019.09.039
Zhao GL, Wu W (2019) Tensor-based type-2 random vector functional link network. In:2019 Chinese Control And Decision Confer-ence (CCDC), pp 3278–3285, https://doi.org/10.1109/CCDC.2019.8833161
Zheng F, Liu ZY, Chen YJ, et al (2020) Anovel adaptive multi-view non-negative graphsemi-supervised ELM. IEEE Access 8:116,350–116,362. https://doi.org/10.1109/ACCESS.2020.2998428
