Patronage

Tommaso Agasisti, Patrizia Falzetti pag. 2 Socioeconomic sorting and test scores:an empirical analysis in the

Italian junior secondary schools

Dario Albarello, Vera D’Amico pag. 9Empirical testing of probabilistic seismic hazardmodels

Federico Andreis, Pier Alda Ferrari pag. 15 A proposal for the multidimensional extension ofCUB models Morten Arendt Rasmussen, Evrim Acar pag. 19Data fusion in the framework of coupled matrix tensor factorization

withcommon, partially common and unique factors Luigi Augugliaro, Angelo M. Mineo pag. 20Estimation of Sparse Generalized LinearModels: the dglars package

Antonio Balzanella, Lidia Rivoli, Elvira Romano pag. 24A comparison between two tools for data streamsummarization

Lucio Barabesi, Giancarlo Diana, Pier Francesco Perri pag. 28Gini Index Estimation in Randomized ResponseSurveys

Francesco Bartolucci, Federico Belotti, Franco Peracchi pag. 32A test for time-invariant individual effects ingeneralized linear models

for panel data

Erich Battistin, Carlos Lamarche, Enrico Rettore pag. 36Identification of the distribution of the causal effect of an intervention

using a generalised factor model Matilde Bini, Lucio Masserini pag. 37Internal effectiveness of educational offer andstudents’ satisfaction: a SEM approach

Matilde Bini, Leopoldo Nascia, Alessandro Zeli pag. 41Groups heterogeneity and sectorsconcentration: a structural equation

modelingfor micro level analysis of firms Giuseppe Boari, Marta Nai Ruscone pag. 45Use of Relevant Principal Components to Definea Simplified

Multivarate Test Procedure ofOptimal Clutering

Giuseppe Boari, Gabriele Cantaluppi, Angelo Zanella pag. 49Some Distance Proposals for Cluster Analysis inPresence of Ordinal

Variables

Laura Bocci, Donatella Vicari pag. 53A general model for INDCLUS with externalinformation

Paola Bongini, Paolo Trivellato, Mariangela Zenga pag. 57The financial literacy and the undergraduates

Riccardo Bramante, Marta Nai Ruscone, Pasquale Spani pag. 61Credit risk measurement and ethical issue: someevidences from

the italian banks

Pierpaolo Brutti, Lucio Ceccarelli, Fulvio De Santis, Stefania Gubbiotti pag. 65On the Stylometric Authorship of Ovid’s DoubleHeroides: An Ensemble Clustering Approach

Silvia Caligaris, Fulvia Mecatti and Patrizia Farina pag. 69Causal Inference in Gender Discrimination inChina:

Nutrition, Health, Care

Giorgio Calzolari, Antonino Di Pino pag. 73Self-Selection and Direct Estimation of Across-Regime Correlation

Parameter

Maria Gabriella Campolo, Antonino Di Pino, Ester Lucia Rizzi pag. 77Modern Vs. Traditional: A cluster-basedspecification of gender and

familistic attitudesand their influence on the division of labour of Italian

couples

Gabriele Cantaluppi, Marco Passarotti pag. 81Clustering the Four Gospels in the Greek,Latin, Gothic and Old Church

SlavonicTranslations

Carmela Cappelli, Francesca Di Iorio pag. 85Regression Trees for change point analysis:methods, applications

and recent developments

Roberto Casarin and Marco Tronzano and Domenico Sartore pag. 89Bayesian Stochastic Correlation Models

Rosalia Castellano, Gennaro Punzo, Antonella Rocca pag. 93Evaluating the selection effect in labour marketswith a low female

participation

Paola Cerchiello, Paolo Giudici pag. 97A statistical based H index for the evaluation ofe-markets

Annalisa Cerquetti pag. 101Bayesian nonparametric estimation of globaldisclosure risk

Enrico Ciavolino, Roberto Savona pag. 105The Forecasting side of Sovereign Risk: aGeneralized Cross Entropy Approach

Nicoletta Cibella, Tiziana Tuoto, Luca Valentino pag. 109What data tell you that models can’t say

Roberto Colombi, Sabrina Giordano pag. 114Multiple Hidden Markov Models for CategoricalTime Series

Pier Luigi Conti, Daniela Marella pag. 118Asymptotics in survey sampling for high entropysampling designs

Claudio Conversano, Massimo Cannas, Francessco Mola pag. 122On the Use of Recursive Partitioning in CasualInference: A Proposal

Franca Crippa, Marcella Mazzoleni, Mariangela Zenga pag. 128Keeping the pace with higher education. A fuzzystates gender study

F. Cugnata, C. Guglielmetti and S. Salini pag. 133CUB model to validate FACIT TS-PSmeasurement instrument

Rosario D’Agata, Venera Tomaselli pag. 137Multilevel Approach in Meta-Analysis of Pre-Election Poll Accuracy

Alfonso Iodice D’Enza and Angelos Markos pag. 141Low-dimensional tracking of associationstructures in categorical data

Giulio D’Epifani pag. 145Self-censored Categorical ResponsesA device for recovering latent

behaviors

Pierpaolo D’Urso, Marta Disegna, Riccardo Massari pag. 150Tourism Market Segmentation with ImpreciseInformation

Utkarsh J. Dang, Salvatore Ingrassia, Paul D. McNicholas and Ryan Browne pag. 154Cluster-weighted models for multivariateresponse and extensions

Cristina Davino, Domenico Vistocco pag. 158Unsupervised Classification through QuantileRegression

F. Marta L. Di Lascio, Simone Giannerini pag. 162A copula-based approach to discoverinter-cluster dependence

relationships

Josè G. Dias, Sofia B. Ramos pag. 166Hierarchical market structure of Euro arearegime dynamics

Drago Carlo, Balzanella Antonio pag. 170Consensus Community Detection: a NonmetricMDS Approach

Fabrizio Durante, Roberta Pappad`a and Nicola Torelli pag. 174Clustering financial time series by measures oftail dependence

Marco Enea, Antonella Plaia pag. 178Influence diagnostics for generalized linearmixed models:

a gradient-like statistic

Enrico Fabrizi, Maria R. Ferrante, Carlo Trivisano pag. 182Joint estimation of poverty and inequalityparameters in small areas

Giorgio Fagiolo, Andrea Roventini pag. 187Macroeconomic Policy in DSGE andAgent-Based Models

Salvatore Fasola, Mariangela Sciandra pag. 191New Flexible Probability Distributions forRanking Data

Maria Brigida Ferraro, Paolo Giordani pag. 195A new fuzzy clustering algorithm with entropyregularization

Camilla Ferretti, Piero Ganugi, Renato Pieri pag. 199Mobility measures for the dairy farms inLombardy

Silvia Figini, Marika Vezzoli pag. 203Model averaging and ensemble methods for riskcorporate estimation

Luis Angel García-Escudero, Alfonso Gordaliza, Carlos Matrán, Agustín Mayo-Iscar pag. 207New proposals for clustering based on trimmingand restrictions

Andreas Geyer-Schulz, Fabian Ball pag. 211Formal Diagnostics for Graph Clustering: TheRole of Graph

Automorphisms

Massimiliano Giacalone, Angela Alibrandi pag. 215An overview on multiple regression models basedon permutation tests

Francesca Giambona, Mariano Porcu pag. 219The determinants of Italian students’ readingscores: a Quantile Regression analysis

Paolo Giordani, Henk A.L. Kiers, Maria Antonietta Del Ferraro pag. 223The R Package ThreeWay

Giuseppe Giordano, Ilaria Primerano pag. 227Co-occurence Network from SemanticDifferential Data

Paolo Giudici pag. 231Financial risk data analysis

Silvia Golia, Anna Simonetto pag. 237A Comparison between SEM and Rasch model:the polytomous case

Anna Gottard pag. 241Some considerations on VCUB models

Francesca Greselin, Salvatore Ingrassia pag. 245Data driven EM constraints for mixtures offactor analyzers

Leonardo Grilli, Carla Rampichini, Roberta Varriale pag. 249Predicting students’ academic performance: achallenging issue in statistical modelling

Luigi Grossi, Fany Nan pag. 255Robust estimation of regime switching models

Kristian Hovde Liland pag. 259Variable selection in sequential multi-block analysis

Maria Iannario pag. 260Robustness issues for a class of models forordinal data

Maria Iannario, Domenico Piccolo pag. 264A class of ordinal data models in R

Salvatore Ingrassia, Antonio Punzo pag. 268Parsimony in Mixtures with Random Covariates

Hiroshi Inoue pag. 272International Relations Based on the VotingBehavior in General

Assembly

Carmela Iorio, Massimo Aria, Antonio D’Ambrosio pag. 276Visual model representation and selection forclassification and

regression trees

Monia Lupparelli, Luca La Rocca, Alberto Roverato pag. 284Log-Mean Linear Parameterizations for SmoothIndependence

Models

Marica Manisera, Paola Zuccolotto pag. 288Nonlinear CUB models

Marica Manisera, Marika Vezzoli pag. 292Finding number of groups using a penalizedinternal cluster

quality index

Daniela Marella, Paola Vicard pag. 296Object-Oriented Bayesian Network to deal withmeasurement error

in household surveys

Angelos Markos, Alfonso Iodice D’Enza, Michel Van de Velden pag. 300Beyond tandem analysis: joint dimensionreduction and clustering in R

F. Martella and M. Alfò pag. 304A biclustering approach for discrete outcomes

Mariagiulia Matteucci, Stefania Mignani, Roberto Ricci pag. 309A Multidimensional IRT approach to analyzelearning achievement

of Italian students

Sabina Mazza pag. 314Extending the Forward Search to theCombination of Multiple

Classifiers: A Proposal

Fulvia Mecatti, M. Giovanna Ranalli pag. 318Plug-in Bootstrap for Sample Survey Data

Alessandra Menafoglio, Matilde Dalla Rosa and Piercesare Secchi pag. 322A BLU Predictor for Spatially DependentFunctional Data of a

Hilbert Space

Maria Adele Milioli, Lara Berzieri, Sergio Zani pag. 326Comparing fuzzy and multidimensional methodsto evaluate

well-being at regional level

Michelangelo Misuraca, Maria Spano pag. 331Comparing text clustering algorithms from amultivariate perspective

Cristina Mollica, Luca Tardella pag. 335Mixture models for ranked data classification

Isabella Morlini, Stefano Orlandini pag. 339Cluster analysis of three-way atmospheric data

Roberto Nardecchia, Roberto Sanzo, Margherita Velucchi, Alessandro Zeli pag. 345Productivity transition probabilities: A microlevel data analysis

for Italian manufacturingsectors (1998-2007)

Andrea Neri, Giuseppe Ilardi pag. 349Interviewers, co-operation and data accuracy: isthere a link?

Akinori Okada, Satoru Yokoyama pag. 353Nonhierarchical Asymmetric Cluster Analysis

Marco Perone Pacifico pag. 357SuRF: Subspace Ridge Finder

Andrea Pagano, Francesca Torti, Jessica Cariboni, Domenico Perrotta pag. 361Robust clustering of EU banking data

Giuseppe Pandolfo, Giovanni C. Porzio pag. 365On depth functions for directional data

Andrea Pastore, Stefano F. Tonellato pag. 369A generalised Silhouette-width measure

Fulvia Pennoni, Giorgio Vittadini pag. 373Hospital efficiency under two competing paneldata models

Alessia Pini, Simone Vantini pag. 377The Interval-Wise Control of the Family-WiseError Rate for Testing

Functional Data

Mariano Porcu, Isabella Sulis pag. 381Detecting differences between primary schools inmathematics and

reading achievement by usingschools added-value measures

of performance

Antonio Punzo, Paul D. McNicholas, Katherine Morris, Ryan P. Browne pag. 387Outlier Detection via Contaminated MixtureDistributions

Emanuela Raffinetti, Pier Alda Ferrari pag. 392New perspectives for the RDI index in socialresearch fields

Monia Ranalli, Roberto Rocci pag. 396Mixture models for ordinal data: a pairwiselikelihood approach

Marco Riani, Andrea Cerioli, Gianluca Morelli pag. 400Issues in robust clustering

Stèphane Robin pag. 404Deciphering and modeling heterogeneity ininteraction networks

Rosaria Romano, Francesco Palumbo pag. 409Partial Possibilistic Regression Path Modeling

Renata Rotondi pag. 413Classsification of composite seismogenic sourcesthrough probabilitic

score indices

Gabriella Schoier pag. 417On Wild Bootstrap and M Unit Root Test

Luca Scrucca pag. 421On the implementation of a parallel algorithmfor variable selection

in model-based clustering

Paolo Sestito pag. 425The Role of Learning Measurement in theGovernance of an

Education System: anOverview of the Issues

John Shawe-Taylor, Blaz Zlicar pag. 430Novelty Detection with Support Vector Machines

Nadia Solaro pag. 431Multidimensional scaling with incompletedistance matrices:

an insight into the problem

Luigi Spezia, Cecilia Pinto pag. 435Markov switching models for high-frequencytime series: flapper

skate’s depth profile as a casestudy

Ralf Stecking, Klaus B. Schebesch pag. 439Data Privacy in Credit Scoring: Evaluating SVMApproaches Based on

Microaggregated Data

Isabella Sulis, Francesca Giambona, Nicola Tedesco pag. 443Analyzing university students’ careers usingMulti-State Models

Luca Tardella, Danilo Alunni Fegatelli pag. 447BBRecap for Bayesian BehaviouralCapture-Recapture Modeling

Cristina Tortora, Paul D. McNicholas, Ryan P. Browne pag. 451Mixtures of generalized hyperbolic factoranalyzers

Giovanni Trovato pag. 455Testing for endogeneity and countryheterogeneity

Joaquin Vanschoren and Mikio L. Braun, Cheng Soon Ong pag. 461Open science in machine learning

Valerio Veglio pag. 465Logistic Regression and Decision Tree:Performance Comparisons in

EstimatingCustomers’ Risk of Churn

Maurizio Vichi pag. 469Robust Two-mode clustering

Vincenzina Vitale pag. 470Hierarchical Graphical Models and ItemResponse Theory

Sara Viviani pag. 474Extending the JM libraRy

Adalbert F.X. Wilhelm pag. 478Visualisations of Classification Tree Models: AnEvaluative Comparison

Data driven EM constraints for mixtures offactor analyzers

Francesca Greselin and Salvatore Ingrassia

Abstract Mixtures of factor analyzers are becoming more and more popular in thearea of model based clustering of high-dimensional data. In this paper we implementa data-driven methodology to maximize the likelihood function in a constrainedparameter space, to overcome the well known issue of singularities and to reducespurious maxima in the EM algorithm. Simulation results and applications to realdata show that the problematic convergence of the EM, even more critical whendealing with factor analyzers, can be greatly improved.Key words: Mixture of Factor Analyzers, Model-Based Clustering, ConstrainedEM algorithm.

1 Introduction and motivation

Finite mixture distributions, dating back to the seminal work of Newcomb and Pear-son, have been receiving a growing interest in statistical modeling all along the lastcentury. Along the lines of Ghahramani and Hilton (1997) we assume that the datahave been generated by a linear factor model with latent variables modeled as Gaus-sian mixtures. Our purpose is to improve the performances of the EM algorithm,giving practical recipes to overcome some of its issues. Following Ingrassia (2004),in this paper we introduce and implement a procedure for the parameters estimationof mixtures of factor analyzers, which maximizes the likelihood function in a con-strained parameter space, having no singularities and a reduced number of spuriouslocal maxima. Within the Gaussian Mixture (GM) model-based approach to densityestimation and clustering, the density of the d-dimensional random variableX of in-terest is modeled as a mixture of a number, say G, of multivariate normal densitiesin some unknown proportions π1, . . .πG,

Francesca GreselinDepartment of Statistics and Quantitative MethodsMilano-Bicocca UniversityVia Bicocca degli Arcimboldi 8 - 20126 Milano (Italy), e-mail: francesca.greselin@unimib.it

Salvatore IngrassiaDepartment of Economics and BusinessUniversity of CataniaCorso Italia 55 - Catania (Italy), e-mail: s.ingrassia@unict.it

1

2 Francesca Greselin and Salvatore Ingrassia

f (x;θ ) =G

∑g=1

πgφd(x;µg,Σ g)

where φd(x;µ ,Σ ) denotes the d-variate normal density function with mean µ andcovariance matrix Σ . Then, we postulate a finite mixture of linear sub-modelsXi = µg+ΛgUig+eig with probability πg (g= 1, . . . ,G) for i= 1, . . . ,n, for the dis-tribution of the full observation vector X, given the (unobservable) latent factors U,whereΛ g is a d×q matrix of factor loadings, the factors U1g, . . . ,Ung areN (0,Iq)distributed independently of the errors eig, which are independentlyN (0,Ψg) dis-tributed, andΨg is a d× d diagonal matrix (g = 1, . . . ,G). We suppose that q < d,which means that q latent factors are jointly explaining the d observable features ofthe statistical units. Under these assumptions, Σ g = ΛgΛ ′

g+Ψg (g = 1, . . . ,G).The parameter vector θMGFA(d,q,G) now consists of the elements of the com-ponent means µg, the Λg, and the Ψg, along with the mixing proportions πg(g= 1, . . . ,G− 1).

2 The likelihood function and the EM algorithm for MGFA

In this section we summarize the main steps of the EM algorithm for mixtures ofFactor analyzers, see e.g. McLachlan et al. (2003) for details. Let xi (i= 1, . . . ,n) de-notes the realization of Xi. Then, the complete-data likelihood function for a sampleX∼of size n can be written as

Lc(θ ;X∼) =

n

∏i=1

G

∏g=1

!φd"xi|ui;µg,Λ g,Ψg

#φq(uig)πg

$zig. (1)

Due to the factor structure of the model, we consider the alternating expectation-conditional maximization (AECM) algorithm, consisting of the iteration of two con-ditional maximizations, until convergence. There is one E-step and one CM-step, al-ternatively i) considering θ 1= {πg,µg, g= 1, . . . ,G}where the missing data are theunobserved group labels Z

%= (z′1, . . . ,z′n) and ii) considering θ 2 = {(Λg,Ψg), g =

1, . . . ,G} where the missing data are the group labels Z and the unobserved latentfactors U = (U11, . . . ,UnG). In the First Cycle, after updating the z(k+1)ig in the E-step, the M-step provides new values for π (k+1)

g ,µ (k+1)g ,n(k+1)g . In the Second Cycle,

after writing the complete data log-likelihood, some algebras lead to the followingestimate of {(Λg,Ψg), g= 1, . . . ,G}

Λ̂ g = S(k+1)g γ(k)′

g [Θ (k)g ]−1 Ψ̂ g = diag

&S(k+1)g − Λ̂gγ(k)g S(k+1)g

',

whereS(k+1)g = (1/n(k+1)g )

n

∑i=1

z(k+1)ig (xi− µ(k+1)g )(xi− µ(k+1)

g )′

γ(k)g =Λ (k)′g (Λ (k)

g Λ (k)′g +Ψ (k)

g )−1 and Θ (k)g = Iq− γ(k)g Λ (k)

g + γ(k)g S(k+1)g γ(k)′

g .

Data driven EM constraints for mixtures of factor analyzers 3

3 Likelihood maximization in constrained parametric spaces

Along the lines of Ingrassia (2004) let us consider the constrained parameter space

Θ c ={(π1, . . . ,πG,µ1, . . . ,µG,Σ 1, . . . ,ΣG) ∈ Rk[1+d+(d2+d)/2] :

πg ≥ 0, π1+ · · ·+πG = 1, a≤ λig ≤ b, g= 1, . . . ,G i= 1, . . . ,d}. (2)

Applying the eigenvalue decomposition to the square d×d matrix ΛgΛ ′g we can

find Γ g and ∆g such that ΛgΛ ′g = Γ g∆gΓ ′

g where Γ g is the orthonormal matrixwhose rows are the eigenvectors of ΛgΛ ′

g and ∆ g = diag(δ1g, . . . ,δdg) is the sorteddiagonal matrix of the eigenvalues of Λ gΛ ′

g, i.e. δ1g ≥ . . .≥ δqg ≥ 0, and δ(q+1)g =· · · = δdg = 0. Further, applying now the singular value decomposition to Λ g, weget Λ g = UgDgV′

g. This yields ΛgΛ ′g = (UgDgV′

g)(VgD′gU′

g) = UgDgD′gU′

g hencediag(δ1g, . . . ,δqg) = diag(d21g, . . . ,d2qg). We can now modify the EM algorithm insuch a way that the eigenvalues of the covariancesΣ g (for g= 1, . . . ,G) are confinedinto suitable ranges. To this aim we exploit the following inequalities

λmin(Λ gΛ ′g+Ψg)≥ λmin(ΛgΛ ′

g)+λmin(Ψ g)≥ aλmax(Λ gΛ ′

g+Ψg)≤ λmax(ΛgΛ ′g)+λmax(Ψg)≤ b

which enforce (2) when imposing the following constraints

d2ig+ψig ≥ a i= 1, . . . ,d (3)

dig ≤(b−ψig i= 1, . . . ,q (4)

ψig ≤ b i= q+ 1, . . . ,d (5)

for g = 1, . . . ,G, where ψig denotes the i-th diagonal entry ofΨg. In particular, wenote that condition (3) reduces to ψig ≥ a for i= (q+ 1), . . . ,d.It is important to remark that the resulting EM algorithm is monotone, once the

initial guess, say Σ 0g, satisfies the constraints. Further, as shown in the case of gaus-sian mixtures in Ingrassia and Rocci (2007), the maximization of the complete log-likelihood is guaranteed. On another note, a data driven method to gauge the boundsa and b is needed.

4 Numerical studies

A brief numerical study is presented here, to compare the performance of the con-strained vs unconstrained EM algorithm. More simulations have been performed,also with real datasets are available in (see Greselin and Ingrassia, 2013). A sampleof N = 150 data has been generated with weights α = (0.3,0.4,0.3)′ with parame-ters such that maxi,gλi(Σ g) = 4.18 . We run 100 times both the unconstrained andthe constrained AECM algorithms (for different values of the constraints a,b) usinga common random initial clusterings.The unconstrained algorithm attains the rightsolution in 24% of cases; summary statistics about the misclassification error, over

4 Francesca Greselin and Salvatore Ingrassia

the 100 runs, are reported in Table 1. To compare how a and b influences the per-formance of the constrained EM, different pairs of values has been considered, andTable 2 shows the more interesting cases. Further results are reported in Figure 1,where the boxplots of the distribution of the misclassification errors show the poorperformance of the unconstrained algorithm compared to its constrained version.For all values of the upper bound b, the third quartile of the misclassification erroris steadily equal to 0. Indeed, for b= 6,10 and 15 we had no misclassification error,while we observed very low and rare misclassification errors only for b = 20 andb= 25 (respectively 3 and 11 not null values, over 100 runs). Moreover, the robust-ness of the results with respect to the choice of the upper constraint is apparent. Adata driven method to select the bounds can be derived from the observed results, byrunning the constrained EM for increasing values of the upper bound, till a decreasein the final likelihood. The value of b before the decrease, observed over a series ofrun, will be chosen as upper value for the constrained estimation.Table 1 Summary statistics for the Misclassif Error over 100 runs of the unconstrained EM alg

min Q1 Q2 Q3 max0% 17% 36% 45.3% 60%

Table 2 Percentage of convergence to the right maximum of the constrained EM algorithm fora= 0.01 and some values of the upper constraint b

b :+∞ 6 10 15 20 2524% 100% 100% 100% 97% 89%

unconstrained b=6 b=10 b=15 b=20 b=25

0.00.2

0.40.6

0.81.0

Fig. 1 Boxplots of the misclassification error. From left to right, boxplots refer to the unconstrainedalgorithm, then to the constrained algorithm, for a= 0.01 and b= 6,10,15,20,25.

ReferencesGhahramani, Z. and Hilton, G. (1997). The EM algorithm for mixture of factoranalyzers. Tech. Rep. CRG-TR-96-1.

Greselin, F. and Ingrassia, S. (2013) Maximum likelihood estima-tion in constrained parameter spaces for mixtures of factor analyzers,http://arxiv.org/abs/1301.1505.

Ingrassia, S. (2004). A likelihood-based constrained algorithm for multivariate nor-mal mixture models. Stat. Meth. & Appl., 13, 151–166.

Ingrassia, S. and Rocci, R. (2007). Constrained monotone em algorithms for finitemixture of multivariate gaussians. Comp. Stat. & Data Anal., 51, 5339–5351.

McLachlan, G., Peel, D., and Bean, R. (2003). Modelling high-dimensional data bymixtures of factor analyzers. Comp. Stat. and Data Anal., 41, 379–388.