FACULTEIT ECONOMIEEN BEDRIJFSKUNDE

HOVENIERSBERG 24B-9000 GENT

Tel. : 32 - (0)9 – 264.34.61Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER

Wrapped Input Selection using Multilayer Perceptrons for

Repeat-Purchase Modeling in Direct Marketing

Stijn Viaene1, Bart Baesens1, Dirk Van den Poel2, Guido Dedene1 & Jan Vanthienen1

1K.U.Leuven, Dept. of Applied Economic Sciences, Naamsestraat 69, B-3000 Leuven, Belgium.

2Ghent University, Dept. of Marketing, Hoveniersberg 24, B-9000 Ghent, Belgium.

June 2001

2001/102

Corresponding author: Stijn Viaene, Dept. of Applied Economic Sciences, Katholieke Universiteit Leuven,Naamsestraat 69, B-3000 Leuven, Belgium. Phone: ++32 (16) 32 68 90, Fax: ++32 (16) 32 67 32, e-mail:Stijn.Viaene@econ.kuleuven.ac.be.Sponsor : Part of this work was carried out in the context of the KBC Insurance Research Chair that was set upin September 1997 as a pioneering research co-operation between the Leuven Institute for Research onInformation Systems (LIRIS) and the KBC bank and insurance group, which is one of the larger super regionalbank and insurance groups in the Benelux (Europe) with head office in Belgium.

D/2001/7012/03

Wrapped Input Selection using Multilayer Perceptrons for

Repeat-Purchase Modeling in Direct Marketing

Stijn Viaene1, Bart Baesens1, Dirk Van den Poel2, Guido Dedene1 & Jan Vanthienen1

1K.U.Leuven, Dept. of Applied Economic Sciences, Naamsestraat 69, B-3000 Leuven, Belgium.

2Ghent University, Dept. of Marketing, Hoveniersberg 24, B-9000 Ghent, Belgium.

Abstract

In this paper, we try to validate existing theory on and develop additional insight into repeat-purchase

behavior in a direct marketing setting by means of an illuminating case study. The case involves the

detection and qualification of the most relevant RFM (Recency, Frequency and Monetary) variables,

using a neural network wrapper as our input pruning method. Results indicate that elimination of

redundant and/or irrelevant inputs by means of the discussed input selection method allows us to

significantly reduce model complexity without degrading the predictive generalization ability. It is

precisely this issue that will enable us to infer some interesting marketing conclusions concerning the

relative importance of the RFM predictor categories and their operationalizations. The empirical findings

highlight the importance of a combined use of RFM variables in predicting repeat-purchase behavior.

However, the study also reveals the dominant role of the frequency category. Results indicate that a

model including only frequency variables still yields satisfactory classification accuracy compared to the

optimally reduced model.

Keywords: direct marketing, multilayer perceptrons, input selection, response modeling, classification.

2

1 Introduction

It need not be emphasized that customer retention is at least as important as customer acquisition in the

current context of competitive markets, not in the least for mail-order companies. Service providers are

finding themselves in mature markets in which they have to switch their marketing efforts from acquiring

new customers towards retention of existing customers (Grant, 1995; Stone, 1996). Customer relations

undoubtedly represent an important opportunity cost: 5% fewer customers may result in profit losses

between 25% and 85% (Reichheld, 1990). The objective of this paper is to validate existing theory on

and develop additional insight into repeat-purchase behavior in a direct mail setting.

The empirical study focuses on the repeat-purchase-incidence. It studies the issue whether or not

an existing customer, i.e. a customer who has bought before, purchases a product listed in the direct mail

catalogue within a limited time frame. As to the choice of the independent variables, the set-up of the

experiment is limited to assessing the predictive importance of several alternative operationalizations of

the traditionally discussed (R)ecency, (F)requency and (M)onetary predictor categories. This choice is

motivated by the fact that most previous research cites them as being most predictive and because they

are internally available at very low cost (Bauer, 1988; Bult, 1995).

For modeling repeat-purchase behavior, several techniques have already been proposed and

operationalized. To date, most research on this topic uses traditional statistical models. Examples

include logit, probit and discriminant analysis. In this paper we use Multilayer Perceptron (MLP) neural

networks as our baseline modeling technique. As universal approximators, MLPs have shown to be very

promising supervised learning tools for modeling non-linear relationships (Bishop, 1995; Chang, 1996;

Lacher, 1995; Mobley, 2000; Piramuthu, 1999; Sharda, 1996). This, especially in situations where one is

confronted with a lack of domain knowledge, which in turn prevents any valid argumentation to be made

concerning model selection bias on the basis of prior knowledge.

The experiment rests upon the application of a step-wise, MLP-based input pruning method to a

carefully gathered, real-life mail-order data set. The case study data consists of a detailed sample of

1,200 data points obtained from a Belgian mail-order company. It is shown that by making use of the

discussed input selection method, model complexity can be significantly reduced without degrading the

predictive generalization ability. It is precisely this issue that will allow us to infer some interesting

3

marketing conclusions concerning the relative importance of the RFM predictor categories. As argued in

Van den Poel (1999), this has never been thoroughly investigated, let alone in the context of connectionist

modeling. Bass and Wind (1995) cite the confidentiality of the data, resulting from the high business

value of information in this area, as one of the major obstacles.

This paper is organized as follows. In Section 2, we provide a concise overview of response

modeling issues in direct marketing and discuss the data set used in the experiments. Section 3 presents

the theoretical underpinnings of the use of MLPs for classification. The basic experimental set-up is

outlined in Section 4. In Section 5 we present the outline of the step-wise, MLP-based input pruning

wrapper method and its application to the purchase-incidence case at hand. Section 6 offers additional

interpretation and a critical contemplation of the results that were obtained.

2 Response modeling in direct marketing

In the first subsection, we briefly elaborate on some response modeling issues typical of direct marketing.

Hereby, we position both the nature of the problem statement and the chosen variable categories, in casu

the response model and the RFM predictors. In subsection 2.2, we comment on the study data sample

used in the experiment reported on in this paper.

2.1 Setting the stage

Cullinan (1977) is generally credited for identifying the three sets of variables most often used in

database marketing: (R)ecency, (F)requency and (M)onetary (Bauer, 1988; Kestnbaum, 1992). Since

then, the literature has accumulated so many uses of these three variable categories, that there is

overwhelming evidence, both from academic studies as from practitioners’ experience, that RFM

variables are among the most important predictor categories for modeling mail-order repeat-purchasing.

However, when browsing the vast amount of literature, it becomes evident that only very limited attention

has been devoted to selecting the right set of operationalizations of RFM variables to include into the

model of mail-order repeat-buying. In fact, most studies do not offer a formal justification of their choice

4

of variables, which is therefore often of an ad-hoc nature. Instead, the focus of these articles lies mostly

on selecting the appropriate modeling technique.

For mail-order response modeling, several alternative problem formulations have been proposed

based on the choice of the dependent variable. The first category is purchase-incidence modeling (Bult,

1993). In this problem formulation, the main question is whether a customer will purchase during the

next mailing period, i.e. one tries to predict the purchase-incidence within a fixed time interval. Other

authors have investigated related problems dealing with both the purchase-incidence and the amount of

purchase in a joint model (Levin, 1998; Van der Scheer, 1998). A third alternative perspective for

response modeling is to model inter-purchase time through survival analysis or (split-) hazard rate models

(Dekimpe, 1997; Van den Poel, 1998) which model whether a purchase takes place together with the

duration of time until a purchase occurs.

This paper focuses on purchase-incidence modeling in a customer retention context. More

specifically, we consider the issue whether an existing customer, i.e. a customer who has bought before,

purchases a product offered by the mail-order company within a restricted time window. This choice is

motivated by the fact that the majority of previous research in the direct marketing literature focuses on

the purchase-incidence problem (Nash, 1994; Zahavi, 1997). Furthermore, this is exactly the setting that

mail-order companies are typically confronted with. They have to decide whether or not a specific mail-

order offering will be sent to a customer during a certain mailing period.

More specifically, we will investigate how a wrapped, step-wise MLP-based input pruning method

can assist in determining which purchase behavior variables may play a pivotal role in predicting repeat-

purchase behavior by mail-order. The adoption of MLPs for modeling purposes is motivated by the fact

that they are flexible, non-parametric modeling techniques, allowing to perform any complex function

mapping with arbitrarily desired accuracy (Hornik, 1989). Moreover, as a direct consequence, their

inherent ability to account for higher-order input interactions and locally predictive inputs makes them an

excellent alternative for exploratory input selection purposes.

5

2.2 Data description

From a major European mail-order company, we obtained Belgian data on past purchase behavior at the

order-line level, i.e. we know when a customer purchased what quantity of a particular product at what

price as part of what order. This allowed us, in close co-operation with domain experts and guided by the

literature to derive all the necessary purchase behavior variables for a total sample size of 1,200

customers. For each customer, these variables were measured in the period between July 1st 1993 and

June 30th 1997. The goal is to predict whether an existing customer will repurchase in the observation

period between July 1st 1997 and December 31st 1997 using the information provided by the purchase

behavior variables. This six-month running period for a catalogue is typical of most European general-

purpose mail-order companies. Our four-year purchase history compares favorably to the six-month

prediction period (Van den Poel, 1999). Of the 1,200 customers, 37.6% of existing customers actually

repurchased during the observation period. We used the customer as unit of analysis (as opposed to the

household) because it corresponds to the level at which the results of the analysis are used, i.e. the

individual customer will receive an individually addressed mail package.

As a form of pre-processing, the few missing values were handled by the unconditional mean

imputation procedure which replaces them by the average of the corresponding input over the whole data

set (Little, 1992).

The Recency, Frequency and Monetary variables have then been modeled as follows. Recency is

operationalized as the number of days since the last purchase (Bauer, 1988). An alternative

operationalization would be the number of consecutive mailings without response (Bult, 1995). As noted

by Kestnbaum (Kestnbaum, 1979) predictor variables may have to be transformed to obtain their full

predictive performance. Hence, we include the "Log" transformation of the recency variable as an

additional input to the MLPs. We motivate the choice of the Log transformation as a means to reduce the

skewness of the distribution of the original recency variable.

Although in most studies no detailed results are reported, the frequency variable is generally

considered to be the most important of the RFM predictors (Nash, 1994). Frequency is usually

operationalized as the number of purchases made in a certain time period (Bauer, 1988; Bult, 1995).

When considering time interval length, inclusion versus exclusion of returned items and order-line versus

6

order level processing, many combinations of these variables are possible. We decide to consider 2 levels

for each factor, which results in a 2x2x2 design (i.e. 8 operationalizations) as indicated in Figure 1. In

the frequency column, “Fr” refers to frequency. “Year” refers to the frequency during the last 12 months

and “Hist” refers to the frequency during the whole customer history. ”NoRet” refers to the fact that

returned items are omitted, whereas ”Returns” refers to the fact that returned merchandise are included in

the count. ”Orderlines” refers to the fact that the frequency reflects a count of the number of order-lines

and ”DiffOrders” refers to the fact that not order-lines but rather the number of different dates (i.e. the

purchase occasions) on which orders are placed are counted.

Monetary value can either be operationalized as (a) the total accumulated monetary amount of

spending by a customer during a certain amount of time (Cullinan, 1977), (b) the highest transaction sale

or (c) the average order size (Nash, 1994). In the monetary column of Figure 1, ”Mon” refers to

monetary value. ”Year” refers to the monetary value accumulated over the last 12 months whereas ”Hist”

refers to the monetary value accumulated over the whole customer history. ”Max” refers to the highest

transaction sale over the whole customer history and ”Avg” refers to the average transaction order size

over the whole customer history. ”NoRet” refers to the fact that returns are deleted before processing.

Most authors do agree on the fact that the monetary value should reflect the total net dollars of past

orders, i.e. sales from past orders minus the dollar value of returns and refunds (Bauer, 1988). Therefore

we do not consider operationalizations including returns. As suggested by Shepard (1997), we again use

the logarithmic transformation of all monetary variables to reduce the skewness in the distribution of

values.

Figure 1: RFM operationalizations included in the data set.

3 Multilayer perceptrons for classification

Neural networks (NNs) have shown to be very promising supervised learning tools for modeling complex

non-linear relationships (Bishop, 1995). They are designed to deal with both regression and classification

tasks. Since regression is beyond the scope of this paper, the discussion will be limited to the

7

classification problematic. As universal approximators, NNs can significantly improve the predictive

accuracy of an inference model compared to statistical techniques that are linear in the model parameters

(Hornik, 1989; Bishop, 1995). A NN is typically composed of an input layer, one or more hidden layers

and an output layer, each consisting of several neurons. Each neuron processes its inputs and generates

one output value that is transmitted to the neurons in the subsequent layer. In a multilayer perceptron

(MLP), all neurons and layers are arranged in a feedforward manner. A three-layer MLP then performs

the following non-linear function mapping:

))xw(fw(fy )m()m(1122= , (1)

where )x,...,x,x(x )m(d

)m()m()m(21= is a d -dimensional input vector corresponding to a specific data

instance [ ]n..m 1∈ that is labeled by a target variable )m(t . 1w and 2w are weight vectors of the hidden

and output layer, respectively, and )m(y is the MLP-produced output vector associated with the thm

data instance. 1f and 2f are termed transfer functions and essentially allow the network to perform

complex non-linear function mappings.

For a binary classification problem, one commonly opts for a three-layer MLP with one output

unit. It is then convenient to use the logistic function, i.e.

)zexp()z(f

−+=

11 (2)

as the transfer function in the output layer ( )f 2 , since its output is limited to a value within the range

[ ]10.. . This allows the output )m(y of the MLP to be interpreted as a bayesian posterior probability

(Bishop, 1995). These probabilities may then be mapped to class labels using a threshold value of e.g.

0.5.

The weight vectors 1w and 2w together make up the parameter vector w , which needs to be

estimated (learned) during a training process. Given a training data set [ ]{ }n..m t,xD )m()m( 1== , the

8

weight vector w of the MLP is randomly initialized and iteratively adjusted so as to minimize an

objective function, typically a sum of squared errors DE , i.e.

( )2

121�

=−=

n

m

)m()m(D ytE . (3)

The backpropagation algorithm performs this minimization by using repeated evaluation of the

gradient DE and the chain rule of derivative calculus. Due to the problems of slow convergence and

relative inefficiency of this algorithm, new and improved optimization methods have been suggested.

The reader is referred to Bishop (1995) for further details. In this paper, we will use the Levenberg-

Marquardt method to minimize the objective function )(F w , i.e.

ww EE)(F D αβ += , (4)

whereby typically �=i

iwE 221

w with i running over all elements of the weight vector w . The

inclusion of wE in )(F w constrains the size of the network weights w and is referred to as

regularization. When the weights are kept small, the network response will be smooth so that the network

is prevented from fitting the noise in the training data (Bishop, 1995; MacKay, 1992).

There is no standard way to determine both parameters α and β . They are often tuned off-line.

We adopt the evidence framework of MacKay (1992) to determine both α and β . In this bayesian

framework, both α and β are interpreted as model parameters and are optimized on-line during the

Levenberg-Marquardt optimization. This approach chooses α and β by maximizing the likelihood

function )H,,D(P βα with H being the functional form of the network. This is done using bayes'

theorem which allows to map all prior assumptions (e.g. probability distributions) into posterior

knowledge. For more mathematical details on bayesian learning, we refer to the work of MacKay (1992).

9

4 Basic experimental set-up

As a pre-processing stage to the training of the MLP, all 18 inputs are statistically normalized to a mean

of zero and a standard deviation of one (Bishop, 1995) according to the following formula:

[ ] [ ] .1..nmand1..di

ix

ixmixm

ix ∈∈−

= ,)(

)(~σ

(5)

All MLPs in this study have one hidden layer, influenced by theoretical works, which show that a

single hidden layer is sufficient to approximate any complex non-linear function with any desired degree

of accuracy (Hornik, 1989). After experimentation within the range [ ]10..1 , the number of hidden units

was set to 3. Both hidden and output units use logistic transfer functions. All analyses were conducted

using the Neural Network 3.0 toolbox of the MatlabTM 5.2 workbench.

Performance is measured by means of the Percentage Correctly Classified instances (PCC) and the

Area under the Receiver Operating Curve (AUROC). The PCC represents the classification accuracy

using a default threshold value of 0.5 to map the bayesian posterior output probabilities of the MLP into

binary class labels. This implicitly assumes equal misclassification costs for false positive and false

negative predictions (Provost, 1998). The Receiver Operating Characteristic (ROC) curve, on the other

hand, is a 2-dimensional graphical illustration of the sensitivity ('true alarms') on the Y-as versus (1-

specificity) ('false alarms') on the X-axis for various values of the classification threshold (decision

criterion) (Egan, 1975; Swets, 1982; Hanley, 1983). Remember that the sensitivity is the percentage of

buyers that are correctly identified by the MLP, whereas the specificity is the percentage of non-buyers

that are correctly identified by the MLP. The ROC curve basically illustrates the behavior of a classifier

without regard to specific misclassification costs. Classifier A is superior to classifier B if the ROC curve

associated with A is situated above that of B. However, both ROC curves may intersect making a

comparison less obvious. This may be overcome by calculating the Area under the Receiver Operating

Curve (AUROC) which is an overall performance measure that has achieved wide acceptance.

10

In our experiments, we use a 10-fold cross-validation procedure in which the data set consisting of

1,200 data instances is split into 10 mutually exclusive folds of equal size. 10-fold cross-validation is a

re-sampling technique in which 10 classifiers, in our case MLPs, are trained each time using only 9 folds

of the data (training folds) and the remaining fold (testing fold) for testing. As a result, all observations

are used for training and each observation is used exactly once for testing. PCCtest (PCCtrain) and

AUROCtest (AUROCtrain) are then computed by averaging the PCC and AUROC on the testing (training)

folds over all 10 sub-experiments of the 10-fold cross-validation procedure.

5 Input selection experiment

5.1 Input selection in a nutshell

Input selection is a commonly adhered technique to reduce model complexity. The goal is to find a

reduced co-ordinate system that allows to project a data sample on a more compact representation. The

general assumption underlying this operation and justifying it, is that the studied data sample

approximately lies within the bounds of this reduced space. As such, models with fewer inputs are

capable of improving both human understanding and computational performance. Moreover, elimination

of redundant and/or irrelevant inputs may then also improve the predictive power of an algorithm

(Bellman, 1961).

Finding the optimal input subset to an induction algorithm from among a multitude of available

predictors is a highly non-trivial problem. An optimal input subset can only be obtained when the input

space is exhaustively searched. When k inputs are present, this would imply the need to evaluate

12 −k input subsets. Unfortunately, as k grows, this very quickly becomes computationally infeasible

(John, 1994). For that reason, a heuristic search procedure through the vast search space is often

preferred.

Input selection can then either be performed as a pre-processing step and independent of the

induction algorithm, or explicitly make use of it. The former approach is termed ‘filter’, the latter

‘wrapper’ (John, 1994). Filter methods operate independently of any learning algorithm. Undesirable

inputs are filtered out of the data before induction commences. Filters typically make use of all the

11

available training data when selecting a subset of inputs. Among the well-known filter approaches are

Focus (Almuallim, 1991) and Relief (Kira, 1992). Wrapper methods make use of the actual target

learning algorithm to evaluate the usefulness of inputs. Typically the input evaluation heuristic that is

used is based upon inspection of the trained parameters and/or comparison of predictive performance

under different input subset contingencies. Input selection is then often performed in a sequential fashion,

e.g. guided by a best-first input selection strategy. The backward selection scheme starts from a full input

set and step-wise prunes input variables that are undesirable. The forward selection scheme starts from

the empty input set and step-wise adds input variables that are desirable. Hybrids of the above also exist.

5.2 Step-wise pruning using a multilayer perceptron wrapper

In this paper, input selection is implemented using a typical wrapper approach with a best-first search

heuristic guiding the backward search procedure towards the optimal input set (John, 1994). Starting with

the full set, all inputs are pruned sequentially, i.e. one by one. We use multilayer perceptron neural

networks as our baseline induction mechanism. Given an initial data set D with d inputs, we proceed in

steps of training and input desirability evaluation. Figure 2 presents the outline of the procedure.

Figure 2: Outline of pruning step k of the backward input selection procedure.

In each step [ ]d..k 1∈ of the backward input selection procedure, a 10-fold cross-validation

experiment is carried out. The desirability (importance) of each input [ ]kF..i 1∈ of the input subset kF

at the start of step k of the pruning scheme is then assessed in two stages. In a first stage an assessment

is made per sub-experiment of the 10-fold cross-validation, giving rise to 10 assessments 1021 iii S,...,S,S

per input i . More specifically, the assessment per sub-experiment is based on the use of (a) the 9 training

data folds of that sub-experiment and (b) the multilayer perceptron trained on the latter data. In a

subsequent stage, a global assessment of the desirability of an input i in pruning step k is made by

aggregating all 10 assessments 1021 iii S,...,S,S .

12

The evaluation of the desirability of the thi input in step k is then operationalized in the

following sensitivity index ( iS ):

�=

=10

1iijSiS , where ),(),( jijDjijDij xExES ww −= . (5)

DE stands for the sum of squared errors as defined in (3). The argument jw represents the trained

weight vector of the MLP for sub-experiment j . The argument ijx represents the thi input and the

argument ijx its average over all training folds of sub-experiment j . The index ijS implements the

concept of sensitivity of the MLP trained in sub-experiment j to the presence/absence of input i . The

input is perturbed to its mean and the impact on the network output is computed in terms of a difference

in DE . This evaluation essentially amounts to a strategy of constant substitution, treating the input to be

neglected as missing by substituting its effect to its mean over the whole sample (Moody, 1991; Moody,

1992; Van De Laar, 1999). Notice that no retraining of the MLP is needed while computing these

sensitivities. Remember, as a result of the 10-fold cross-validation set-up, each input i is characterized

by 10 sensitivity indices, 1021 iii S,...,S,S , one for each of the 10 sub-experiments. These ijS are

summed per input i over all 10 sub-experiments, giving rise to iS . The input p with the lowest

aggregated sensitivity index is then removed from further consideration in the next steps of the backward

selection procedure.

The concept of sensitivity of the model to the presence/absence of an input, as defined by the

above sensitivity measure in (5), does not completely correspond to the concept of causal relevance of an

input within the real, but unknown functional relationship. Interaction and correlation effects among

inputs tend to obscure a rightful assessment of the causal relevance of an input. However, the step-wise

nature of the input selection approach partially counters this. The way interaction effects are accounted

for, is illustrated in the following example. Suppose two inputs are interacting in a significant way.

Setting either one of these inputs to their mean will, by definition, destroy the interaction resulting in a

large value for the aggregated sensitivity index of that input. Correlation effects are coped with according

13

to the following rationale. Consider the situation where two inputs are nearly perfectly correlated and at

least one of them has an inherent significant causal contribution within the underlying functional

relationship. At first sight, the MLP will seem individually insensitive to either one of these inputs since

setting either one of them to their mean causes little information loss. Both inputs will show low

aggregated sensitivity and be considered for elimination. Suppose that in step k , one of these inputs, i.e.

per definition the one with the lowest aggregated sensitivity, is to be omitted. Due to the re-training of

the MLPs in step 1+k , the aggregated sensitivity of the other input will rise, thus clearly indicating its

significance.

Figure 3 provides an indication of the mean PCC and mean AUROC, averaged over all 10 sub-

experiments of the 10-fold cross-validation, for each step of the pruning procedure. Notice that the MLPs

in step 19 have only the bias term as their input, yielding a mean PCC of approximately 62.4%, which

equals to the majority prediction of non-buyers.

Figure 3: Mean training and test performance at each pruning step.

5.3 Determining the pruned input set

After having discussed the mechanics of the input pruning procedure, the question naturally arises

where to situate the cut-off in order to determine the pruned input set. In deciding how much inputs to

prune, a trade-off between model complexity and model accuracy must be evaluated, also referred to as

the bias/variance trade-off (Friedman, 1997; Geman, 1992). Pruning more inputs results in more compact

and more efficient models, but at the potential cost of a loss in predictive effectiveness. In the literature,

several criteria (heuristics) have been devised to effectively cope with this model selection problem,

among which the Network Information Criterion (Murata, 1994) and the Akaike Information Criterion

(Akaike, 1974).

In this paper, we will determine the cut-off point by means of a series of statistical hypothesis

tests. The procedure is fairly straightforward and makes use of the step-wise nature and cross-validation

set-up discussed in the previous sections. We start by identifying the top of the mean PCCtrain curve

14

which depicts the percentage correctly classified training instances averaged over all 10 sub-experiments

for each step of the pruning procedure.

Naive reasoning would then go for the input set at this point as the pruned input set. For the RFM

case at hand, this would lay the cut-off at pruning step 1. The resulting 'reduced' input space would then

consist of all inputs. However, the cut-off decision would then be purely based on a mean performance

criterion evaluated on the training data. In order to take into account the beneficial effect of reduced

model complexity, we proceed with a sequence of one-tailed t-tests in the following manner.

In subsequent steps, we move along the mean PCCtrain curve, starting at pruning step 1 (i.e.

maximum mean PCCtrain) and perform a series of one-tailed t-tests to determine the point at which the

mean PCCtrain value decreases significantly (5% significance level) vis-à-vis the starting point i.e. pruning

step 1. This procedure allows taking into account the variance of the PCCtrain values over all 10 cross-

validation sub-experiments per pruning step. Using this procedure for the RFM-case at hand, the cut-off

is situated at pruning step 13. The pruned input set then consists of 6 inputs.

5.4 Discussion

It has to be clear that the wrapper procedure we presented in the previous subsections, is in fact a

generic 'meta-level' input selection scheme in the sense that basically any classification mechanism could

be plugged in to operationalize the sensitivity based input assessment. As mentioned in previous

sections, we opted for the use of multilayer perceptron neural networks as our baseline classification

mechanism. This choice is motivated by the fact that MLPs are flexible, non-parametric modeling

techniques, allowing to perform any complex function mapping with arbitrarily desired accuracy (Hornik,

1989). They are among the best-suited techniques to account for higher-order input interactions and

locally predictive inputs, which makes them an excellent choice for exploratory input selection purposes.

Table 1 depicts the results of applying the input selection procedure described above to the RFM

case at hand. Both full and reduced model results are presented in terms of PCC and AUROC.

Table 1: Mean results contrasted for full and reduced model.

15

Observe from Table 1 how the suggested input selection procedure allows to significantly reduce

the model complexity (from 18 to 6 inputs) without degrading the generalization behavior in terms of test

set performance for both the PCC and AUROC. This clearly illustrates the usefulness of input selection

for the RFM case. The convergent trend for both performance criteria provides additional confidence in

the presented findings.

The order in which the inputs are pruned is given in Table 2.

Table 2: Order of input removal of the step-wise pruning procedure.

Some interesting marketing conclusions can also be inferred. Among the 6 remaining inputs of the

reduced model, predictors of all three input categories (Recency, Frequency and Monetary) are

encountered. This clearly suggests that the combined use of all three RFM variable categories yields the

richest model for repeat-purchase behavior. Notice the presence of the “Log(Recency)” input which

clearly confirms that reducing the skewness of the Recency input by means of a logarithmic

transformation augments its predictive capability (Kestnbaum, 1979). It must also be remarked that an

input set consisting of only 4 inputs, in casu with only frequency variables, (pruning step 15 in Table 2)

still yields a mean PCCtest of about 71.5% and a mean AUROCtest of about 75%.

This clearly illustrates the importance of the frequency variables in predicting mail-order repeat-

purchase behavior. This piece of empirical evidence supports the hypothesis that the frequency variable

is to be considered the most important of the RFM predictors (Nash, 1994). Moreover, it shows that

including four alternative operationalizations of the frequency variable results in very high predictive

performance. It highlights that not only recent purchase history data (as indicated by the "Year"

specification), but also cumulative information over the past four years (as indicated by the "Hist"

specification) is relevant for predicting future repeat-purchasing behavior. A similar conclusion holds for

the inclusion or exclusion of returned merchandise (“Returns” vs. “NoRet”) and whether orderline level

data or data on order counts should be used (“Orderlines” vs. “DiffOrders”). The possibility of using

several operationalizations of the same variable category has been overlooked in most existing research,

which in most of the cases only includes one typical operationalization per R, F and M variable.

16

6 Summary and conclusions

In this paper, we studied a wrapped neural network input selection method in a direct marketing setting

by means of an illuminating case study. The case involved the detection and qualification of the most

relevant RFM (Recency, Frequency and Monetary) inputs. Results indicate that elimination of

redundant/irrelevant inputs by means of the discussed input selection approach allows to significantly

reduce model complexity without degrading generalization ability. It is precisely this element that allows

to infer some interesting marketing conclusions concerning the relative importance of the RFM predictor

categories. The empirical findings highlight the importance of a combined use of all three variable

categories in predicting mail-order repeat-purchase behavior. However, the results also illustrate the

dominant role of the frequency variable. Even a model with only frequency variables still yields

satisfactory classification performance when compared to the optimally reduced model. Moreover, we

show that the use of alternative operationalizations for the same variable category (in particular for the

frequency category) is a fruitful pursuit in terms of obtaining higher predictive performance.

7 Bibliography

Akaike H. 1974. A New Look at the Statistical Model Identification. IEEE Transactions on Automatic

Control AC-19: 716-723.

Almuallim H, Dietterich TG. 1991. Learning with many irrelevant features. In Proceedings of the Ninth

National Conference on Artificial Intelligence; 547-552.

Bass FM, Wind J. 1995. Introduction to the special issue: empirical generalisations in marketing. Marketing

Science. 14.(G1-G6).

Bauer A. 1988. A direct mail customer purchase model. Journal of Direct Marketing. 2(3): 16-24.

Bellman R. 1961. Adaptive Control Processes: A Guided Tour. Princeton University Press.

Bishop CM. 1995. Neural networks for pattern recognition. Oxford University Press.

Bult JR, Wansbeek TJ. 1995. Optimal selection for direct mail. Marketing Science. 14: 378-394.

Bult JR. 1993. Target selection for direct marketing. Phd. Dissertation. Groningen University.

17

Chang LK, Han I, Kwon Y. 1996. Hybrid neural network models for bankruptcy predictions. Decision

Support Systems. 18(1): 63-72.

Cullinan GJ. 1977. Picking them by their batting averages’ recency-frequency-monetary method of

controlling circulation. Manual release 2103. Direct Mail/Marketing Association. N.Y.

Dekimpe MG, Degraeve Z. 1997. The attrition of volunteers. European Journal of Operational Research. 98:

37-51.

Egan JP. 1975. Signal Detection Theory and ROC analysis, Series in Cognition and Perception, Academic

Press, New York.

Friedman J. 1997. On Bias, Variance, 0/1 Loss, and the Curse of Dimensionality. Data Mining and

Knowledge Discovery 1: 55-77.

Geman S, Bienenstock E, Doursat R. 1992. Neural Networks and the bias/variance dilemma. Neural

Computation 4: 1-58.

Grant AWH, Schlesinger LA. 1995. Realize Your Customers’ Full Profit Potential. Harvard Business

Review. Sept-Oct: 59-72.

Hanley JA, McNeil BJ. 1983. The meaning and use of the area under a receiver operating characteristic (roc)

curve. Radiology 148: 839-843.

Hornik K, Stinchcombe M, White H. 1989. Multilayer Feedforward Networks are Universal Approximators.

Neural Networks 2: 359-366.

John GW, Kohavi R, Pfleger K. 1994. Irrelevant Inputs and the Subset Selection Problem. In Machine

Learning: Proceedings of the Eleventh International Conference; 121-129.

Kestnbaum RD. 1979. Customer record predictive model for list segmentation, Manual Release 620.2, Direct

Marketing Association, New York.

Kestnbaum RD. 1992. Quantitative database methods. The Direct Marketing Handbook. Nash EL; 588-597.

Kira K, Rendell LA. 1992. The Feature Selection Problem: Traditional Methods and a New Algorithm. In

Proceedings of AAAI 1992; 129-134.

Lacher RC, Coats PK, Sharma Shanker C, Fant LF. 1995. A neural network for classifying the financial

health of a firm. European Journal of Operational Research. 85(1): 53-65.

Levin N, Zahavi J. 1998. Continuous predictive modeling: a comparative analysis. Journal of Interactive

Marketing. 12(2): 5-22.

18

Little RJA. 1992. Regression with missing x’s: a review. Journal of the American Statistical Association.

87(420): 1227-1237.

MacKay DJC. 1992. Bayesian Methods for Adaptive Models, Phd. Dissertation, Computation and Neural

Systems, California Institute of Technology, Pasadena, CA.

Mobley BA, Schechter E, Moore WE, McKee PA, Eichner JE. 2000. Predictions of coronary artery stenosis

by artificial neural network. Artificial Intelligence in Medicine. 18(3): 187-203.

Moody J, Utans J. 1992. Principled Architecture Selection for Neural Networks: Application to Corporate

Bond Rating Prediction. NIPS4; 683-690.

Moody J. 1991. Note on Generalisation, Regularization and Architecture Selection. In Nonlinear Learning

Systems: First IEEE-SP Workshop on Neural Networks for Signal Processing; 1-10.

Murata N, Yoshizawa S, Amari S. 1994. Network Information Criterion - Determining the Number of

Hidden Units for an Artificial Neural Network Model. IEEE Transactions on Neural Networks. 5: 865-

872.

Nash EL. 1994. Direct marketing: strategy, planning, execution. 3rd edition. McGraw-Hill. New York.

Piramuthu S. 1999. Financial credit-risk evaluation with neural and neurofuzzy systems. European Journal of

Operational Research. 112(2): 310-321.

Provost F, Fawcett T, Kohavi R. 1998. The case against accuracy estimation for comparing classifiers. In

Proceedings of the Fifteenth International Conference on Machine Learning; 445-553.

Reichheld FF, Sasser WE. 1990. Zero Defections: Quality comes to Service. Harvard Business Review. Sept-

Oct: 301-307.

Sharda R, Wilson R. 1996. Neural network experiments in business failures prediction: A review of

predictive performance issues. International Journal of Computational Intelligence and Organizations.

1(2): 107-117.

Shepard D. 1997. A primer on statistical modeling for database marketers. In Proceedings of the 19th NCDM

Conference; 83-170.

Stone M, Woodcock N. 1996. Relationship Marketing. Kogan Page.

Swets JA., Pickett RM. 1982. Evaluation of Diagnostic Systems: Methods from Signal Detection Theory,

Academic Press, New York.

19

Van De Laar P, Heskens T, Gielen S. 1999. Partial Retraining: A New Approach to Input Relevance

Determination. International Journal of Neural Systems. 9(1): 75-85.

Van den Poel D, Leunis J. 1998. Database marketing modeling for financial services using hazard rate

models. International Review of Retail, Distribution and Consumer Research. 8 (2): 243-257.

Van den Poel D. 1999. Response Modeling for Database Marketing using Binary Classification. Phd.

Dissertation. K.U. Leuven. Faculty of Economic and Applied Economic Sciences.

Van der Scheer HR. 1998. Quantitative approaches for profit maximization in direct marketing. Phd.

Dissertation. Groningen University.

Zahavi J, Levin N. 1997. Issues and problems in applying neural computing to target marketing. Journal of

Direct Marketing. 11(4): 63-75.

Mean Results Full Model Reduced Model

PCCtrain 74.03% 73.30%

PCC test 71.33% 71.83%

AUROCtrain 78.14% 77.25%

AUROCtest 73.72% 74.68%

Inputs 18 6

Table 1: Mean results contrasted for full and reduced model.

20

Pruning Step Input Pruning Step Input

1 FrHistNoRetOrderlines 10 FrYearNoRetDiffOrders

2 Recency 11 Log(MonAvgNoRet)

3 FrHistReturnsDiffOrders 12 Log(MonYearNoRet)

4 FrHistReturnsOrderlines 13 MonAvgNoRet

5 Log(MonHistNoRet) 14 Log(Recency)

6 MonYearNoRet 15 FrYearReturnsOrderlines

7 Log(MonMaxNoRet) 16 FrYearReturnsDiffOrders

8 MonMaxNoRet 17 FrYearNoRetOrderlines

9 MonHistNoRet 18 FrHistNoRetDiffOrders

Table 2: Order of input removal of the step-wise pruning procedure.

21

Recency Frequency Monetary

- Recency - FrYearNoRetOrderlines - MonYearNoRet

- Log(Recency) - FrYearNoRetDiffOrders - MonHistNoRet

- FrYearReturnsOrderlines - MonMaxNoRet

- FrYearReturnsDiffOrders - MonAvgNoRet

- FrHistNoRetOrderlines - Log(MonYearNoRet)

- FrHistNoRetDiffOrders - Log(MonHistNoRet)

- FrHistReturnsOrderlines - Log(MonMaxNoRet)

- FrHistReturnsDiffOrders - Log(MonAvgNoRet)

Figure 1: RFM operationalizations included in the data set.

22

Figure 2: Outline of pruning step k of the backward input selection procedure.

: suinputstartkF

PRUNING STEP [ ]1..dk ∈

: ininputsofnumberk

F

[ ]..1 : ininputthiofindexkFi ∈

[ ] 10

: �=

=∈∀j

iSkF1..i

(minarg\1���

= =+i

ssinputkk FF

Testing fold

Training fold

: experibsuthjiniinputofysensitivitijS −

experimsub10i.e.,validationcrossfold10 −−

[1.i ∈∀validCross

23

Figure 3: Mean training and test performance at each pruning step.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

Pruning Step

Mea

n A

URO

C

Mean AUROCtrain and AUROCtest Curves

Test

Train

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190.62

0.64

0.66

0.68

0.7

0.72

0.74

0.76

Mean PCCtrain and PCCtest Curves

Train

Test

Mea

n PC

C

Pruning Step

FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 9000 GENT Tel. : 32 - (0)9 – 264.34.61

Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER SERIES 1

94/01 L. GOUBERT, E. OMEY, The long-term labour demand and the role of productivity in manufacturing in eightOECD-countries, June 1994, 24 p.

94/02 F. HEYLEN, Social, economic and political institutions and taxes on labour, September 1994, 38 p. (published inWeltwirtschaftliches Archiv, 1995).

94/03 P. JOOS, H. OOGHE, Comparison between market determined and accounting determined measures of risk : anempirical assessment for the non-financial firms listed on the Brussels stock exhange, October 1994, 35 p.

94/04 R. VANDER VENNET, Market structure and operational efficiency a determinants of EC bank profitability,September 1994, 25 p. (published in Journal of International Financial Markets, Institutions and Money, 1994).

94/05 S. MANIGART, B. CLARYSSE, K. DEBACKERE, Entry despite the network : exploring the relationship betweennetwork structure and entry patterns in emergent organizational populations, December 1994, 39 p.

95/06 G. VAN HUFFEL, P. JOOS, H. OOGHE, Semi-annual earnings announcements and market reaction : some recentfindings for a small capital market, February 1995, 23 p. (published in European Accounting Review, 1996).

95/07 H. SAPIENZA, S. MANIGART, W. VERMEIR, A comparison of venture capitalist governance and value-added inthe U.S. and Western Europe, February 1995, 31 p. (published in Journal of Business Venturing, 1996).

95/08 F. HEYLEN, L. GOUBERT, E. OMEY, Unemployment in Europe : a problem of relative or aggregate demandshocks ? , March 1995, 16 p. (published in International Labour Review, 1996).

95/09 R. VANDER VENNET, The effect of mergers and acquisitions on the efficiency and profitability of EC creditinstitutions, April 1995, 35 p. (published in Journal of Banking and Finance, 1996).

95/10 P. VAN KENHOVE, A comparison between the "pick any" method of scaling and the semantic differential, April1995, 14 p.

95/11 K. SCHOORS, Bad loans in transition economies, May 1995, 42 p.

95/12 P. JOOS, H. OOGHE, Problemen bij het opstellen van classificatiemodellen : een toepassing op commerciëlekredietscoring bij financiële instellingen, Juni 1995, 39 p. (gepubliceerd in Tijdschrift voor Economie enManagement, 1998).

95/13 I. DE BEELDE, The evolution of industrial accounting thought in Belgium in the first half of the 20th century. Atextbook approach, July 1995, 29 p.

95/14 D. SCHOCKAERT, Japanse laagconjunctuur en vastgoedmarktontwikkelingen, Oktober 1995, 24 p. (gepubliceerdin Maandschrift Economie, 1996).

95/15 P. GEMMEL, R. VAN DIERDONCK, The design of a MRP-based hospital service requirements planning system :the impact of different sources of uncertainty, October 1995, October 1995, 23 p.

95/16 J. MATON, The Cape of Good Hope. Employment and income distribution in South Africa, September 1995,October 1995, 59 p.

96/17 D. WAEYTENS, Activity-based information in budgeting : the impact on information asymmetry, budget slackcreation and related dysfunctional behaviors - a lab experiment, February 1996, 40 p.

96/18 R. SLAGMULDER, Using management control systems to achieve alignment between strategic investmentdecisions and strategy, February 1996, 36 p. (published in Management Accounting Research, 1997).

96/19 N. VALCKX, W. DE VIJLDER, Monetary policy and asset prices : a comparison of the Fed's announcementpolicies 1987-1995, March 1996, 19 p. (published in Bank- en Financiewezen, 1996).

96/20 S. VANDORPE, J. DENYS, E. OMEY, De arbeidsmarktintegratie van afgestudeerden uit TSO en BSO : eenlongitudinale studie, Mei 1996, 21 p. (gepubliceerd in Economisch en Sociaal Tijdschrift, 1997)

FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 9000 GENT Tel. : 32 - (0)9 – 264.34.61

Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER SERIES 2

96/21 N. VALCKX, Business cycle properties of financial indicators in Germany, October 1996, 29 p.

96/22 T. TERMOTE, De arbeidsmarktparticipatie van de vrouw, ontwikkeling van de dienstensector en werkgelegenheid,November 1996, 35 p.

97/23 M. VERHUE, Demand for unemployment insurance : a survey-based analysis, January 1997, 25 p.

97/24 R. VAN HOVE, R. FRAMBACH, P. VAN KENHOVE, The impact of physical attractiveness in advertising onconsumer attitude : the role of product involvement, January 1997, 25 p.

97/25 I. DE BEELDE, Creating a profession 'out of nothing'. The case of the Belgian auditing profession, February 1997,27 p.

97/26 L. GOUBERT, De flexibiliteit van de Belgische relatieve lonen, Maart 1997, 27 p.

97/27 S. MANIGART, K. DE WAELE, M. WRIGHT, K. ROBBIE, Venture capitalist's appraisal of investment projects : anempirical study in four European countries, March 1997, 18 p. (published in Entrepreneurship Theory & Practice,1997).

97/28 P. DE PELSMACKER, J. VAN DEN BERGH, Advertising content and irritation. A Study of 226 TV commercials,April 1997, 27 p. (published in Journal of International Consumer Marketing, 1998).

97/29 R. VANDER VENNET, Determinants of EU bank takeovers : a logit analysis, April 1997, 23 p. (published as‘Causes and consequences of EU bank takeovers’, in S. Eijffinger, K. Koedijk, M. Pagano and R. Portes (eds.), TheChanging European Financial Landscape, CEPR, 1999).

97/30 R. COOPER, R. SLAGMULDER, Factors influencing the target costing process : lessons from Japanese practice,April 1997, 29 p.

97/31 E. SCHOKKAERT, M. VERHUE, E. OMEY, Individual preferences concerning unemployment compensation :insurance and solidarity, June 1997, 24 p.

97/32 F. HEYLEN, A contribution to the empirical analysis of the effects of fiscal consolidation : explanation of failure inEurope in the 1990s, June 1997, 30 p. (revised version, co-authored by G. Everaert, published in Public Choice,2000).

97/33 R. FRAMBACH, E. NIJSSEN, Industrial pricing practices and determinants, June 1997, 33 p. (published in D.Thorne Leclair and M. Hartline (eds.), Marketing theory and applications, vol. 8, Proceedings AMA WinterConference 1997).

97/34 I. DE BEELDE, An exploratory investigation of industry specialization of large audit firms, July 1997, 19 p.(published in International Journal of Accounting, 1997).

97/35 G. EVERAERT, Negative economic growth externalities from crumbling public investment in Europe : evidencebased on a cross-section analysis for the OECD-countries, July 1997, 34 p.

97/36 M. VERHUE, E. SCHOKKAERT, E. OMEY, De kloof tussen laag- en hooggeschoolden en de politiekehoudbaarheid van de Belgische werkloosheidsverzekering : een empirische analyse, augustus 1997, 30 p.(gepubliceerd in Economisch en Sociaal Tijdschrift, 1999).

97/37 J. CROMBEZ, R. VANDER VENNET, The performance of conditional betas on the Brussels Stock exchange,September 1997, 21 p. (published in Tijdschrift voor Economie en Management, 2000).

97/38 M. DEBRUYNE, R. FRAMBACH, Effective pricing of new industrial products, September 1997, 23 p. (published inD. Grewal and C. Pechmann (eds.), Marketing theory and applications, vol. 9, Proceedings AMA WinterConference 1998).

97/39 J. ALBRECHT, Environmental policy and the inward investment position of US 'dirty' industries, October 1997,20 p. (published in Intereconomics, 1998).

FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 9000 GENT Tel. : 32 - (0)9 – 264.34.61

Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER SERIES 3

97/40 A. DEHAENE, H. OOGHE, De disciplinering van het management : een literatuuroverzicht, oktober 1997, 28 p.(published in Economisch en Sociaal Tijdschrift, 2000).

97/41 G. PEERSMAN, The monetary transmission mechanism : empirical evidence for EU-countries, November 1997, 25p.

97/42 S. MANIGART, K. DE WAELE, Choice dividends and contemporaneous earnings announcements in Belgium,November 1997, 25 p. (published in Cahiers Economiques de Bruxelles, 1999).

97/43 H. OOGHE, Financial Management Practices in China, December 1997, 24 p. (published in European BusinessReview, 1998).

98/44 B. CLARYSSE, R. VAN DIERDONCK, Inside the black box of innovation : strategic differences between SMEs,January 1998, 30 p.

98/45 B. CLARYSSE, K. DEBACKERE, P. TEMIN, Innovative productivity of US biopharmaceutical start-ups : insightsfrom industrial organization and strategic management, January 1998, 27 p. (published in International Journal ofBiotechnology, 2000).

98/46 R. VANDER VENNET, Convergence and the growth pattern of OECD bank markets, February 1998, 21 p.(forthcoming as ‘The law of proportionate effect and OECD bank sectors’ in Applied Economics, 2001).

98/47 B. CLARYSSE, U. MULDUR, Regional cohesion in Europe ? The role of EU RTD policy reconsidered, April 1998,28 p. (published in Research Policy, 2000).

98/48 A. DEHAENE, H. OOGHE, Board composition, corporate performance and dividend policy, April 1998, 22 p.

98/49 P. JOOS, K. VANHOOF, H. OOGHE, N. SIERENS, Credit classification : a comparison of logit models anddecision trees, May 1998, 15 p.

98/50 J. ALBRECHT, Environmental regulation, comparative advantage and the Porter hypothesis, May 1998, 35 p.(published in International Journal of Development Planning Literature, 1999)

98/51 S. VANDORPE, I. NICAISE, E. OMEY, ‘Work Sharing Insurance’ : the need for government support, June 1998,20 p.

98/52 G. D. BRUTON, H. J. SAPIENZA, V. FRIED, S. MANIGART, U.S., European and Asian venture capitalists’governance : are theories employed in the examination of U.S. entrepreneurship universally applicable ?, June1998, 31 p.

98/53 S. MANIGART, K. DE WAELE, M. WRIGHT, K. ROBBIE, P. DESBRIERES, H. SAPIENZA, A. BEEKMAN,Determinants of required return in venture capital investments : a five country study, June 1998, 36 p. (forthcomingin Journal of Business Venturing, 2001)

98/54 J. BOUCKAERT, H. DEGRYSE, Price competition between an expert and a non-expert, June 1998,29p. (published in International Journal of Industrial Organisation, 2000).

98/55 N. SCHILLEWAERT, F. LANGERAK, T. DUHAMEL, Non probability sampling for WWW surveys : a comparison ofmethods, June 1998, 12 p. (published in Journal of the Market Research Society, 1999).

98/56 F. HEYLEN. Monetaire Unie en arbeidsmarkt : reflecties over loonvorming en macro-economisch beleid, juni 1998,15 p. (gepubliceerd in M. Eyskens e.a., De euro en de toekomst van het Europese maatschappijmodel, Intersentia,1999).

98/57 G. EVERAERT, F. HEYLEN, Public capital and productivity growth in Belgium, July 1998, 20 p. (published inEconomic Modelling, 2001).

98/58 G. PEERSMAN, F. SMETS, The Taylor rule : a useful monetary policy guide for the ECB ?, September 1998, 28 p.(published in International Finance, 1999).

FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 9000 GENT Tel. : 32 - (0)9 – 264.34.61

Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER SERIES 4

98/59 J. ALBRECHT, Environmental consumer subsidies and potential reductions of CO2 emissions, October 1998, 28 p.

98/60 K. SCHOORS, A payment system failure and its consequences for interrepublican trade in the former Soviet Union,December 1998, 31 p.

98/61 M. DE LOOF, Intragroup relations and the determinants of corporate liquid reserves : Belgian evidence, December1998, 29 p. (published in European Financial Management, 2000).

98/62 P. VAN KENHOVE, W. VAN WATERSCHOOT, K. DE WULF, The impact of task definition on store choice andstore-attribute saliences, December 1998, 16 p. (published in Journal of Retailing, 1999).

99/63 P. GEMMEL, F. BOURGONJON, Divergent perceptions of TQM implementation in hospitals, January 1999, 25 p.(forthcoming in Journal of Management in Medicine, 2000)

99/64 K. SCHOORS, The credit squeeze during Russia's early transition. A bank-based view, January 1999, 26 p.

99/65 G. EVERAERT, Shifts in balanced growth and public capital - an empirical analysis for Belgium, March 1999, 24 p.

99/66 M. DE LOOF, M. JEGERS, Trade Credit, Corporate Groups, and the Financing of Belgian Firms, March 1999, 31 p.(published in Journal of Business Finance and Accounting, 1999).

99/67 M. DE LOOF, I. VERSCHUEREN, Are leases and debt substitutes ? Evidence from Belgian firms, March 1999,11 p. (published in Financial Management, 1999).

99/68 H. OOGHE, A. DEHAENE, De sociale balans in België : voorstel van analysemethode en toepassing op hetboekjaar 1996, April 1999, 28 p. (gepubliceerd in Accountancy en Bedrijfskunde Kwartaalschrift, 1999).

99/69 J. BOUCKAERT, Monopolistic competition with a mail order business, May 1999, 9 p. (published in EconomicsLetters, 2000).

99/70 R. MOENAERT, F. CAELDRIES, A. LIEVENS, E. WOUTERS, Communication flows in international productinnovation teams, June 1999, p. (published in Journal of Product Innovation Management, 2000).

99/71 G. EVERAERT, Infrequent large shocks to unemployment. New evidence on alternative persistence perspectives,July 1999, 28 p.

99/72 L. POZZI, Tax discounting and direct crowding-out in Belgium : implications for fiscal policy, August 1999, 21 p.

99/73 I. VERSCHUEREN, M. DE LOOF, Intragroup debt, intragroup guaranties and the capital structure of Belgian firms,August 1999, 26 p.

99/74 A. BOSMANS, P. VAN KENHOVE, P. VLERICK, H. HENDRICKX, Automatic Activation of the Self in a PersuasionContext , September 1999, 19 p. (forthcoming in Advances in Consumer Research, 2000).

99/75 I. DE BEELDE, S. COOREMAN, H. LEYDENS, Expectations of users of financial information with regard to thetasks carried out by auditors , October 1999, 17 p.

99/76 J. CHRISTIAENS, Converging new public management reforms and diverging accounting practices in Belgian localgovernments, October 1999, 26 p. (forthcoming in Financial Accountability & Management, 2001)

99/77 V. WEETS, Who will be the new auditor ?, October 1999, 22 p.

99/78 M. DEBRUYNE, R. MOENAERT, A. GRIFFIN, S. HART, E.J. HULTINK, H. ROBBEN, The impact of new productlaunch strategies on competitive reaction in industrial markets, November 1999, 25 p.

99/79 H. OOGHE, H. CLAUS, N. SIERENS, J. CAMERLYNCK, International comparison of failure prediction modelsfrom different countries: an empirical analysis, December 1999, 33 p.

FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 9000 GENT Tel. : 32 - (0)9 – 264.34.61

Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER SERIES 5

00/80 K. DE WULF, G. ODEKERKEN-SCHRÖDER, The influence of seller relationship orientation and buyer relationshipproneness on trust, commitment, and behavioral loyalty in a consumer environment, January 2000, 27 p.

00/81 R. VANDER VENNET, Cost and profit efficiency of financial conglomerates and universal banks in Europe.,February 2000, 33 p. (forthcoming in Journal of Money, Credit, and Banking, 2001)

00/82 J. BOUCKAERT, Bargaining in markets with simultaneous and sequential suppliers, April 2000, 23 p. (forthcomingin Journal of Economic Behavior and Organization, 2001)

00/83 N. HOUTHOOFD, A. HEENE, A systems view on what matters to excel, May 2000, 22 p.

00/84 D. VAN DE GAER, E. SCHOKKAERT, M. MARTINEZ, Three meanings of intergenerational mobility, May 2000, 20p. (forthcoming in Economica, 2001)

00/85 G. DHAENE, E. SCHOKKAERT, C. VAN DE VOORDE, Best affine unbiased response decomposition, May 2000,9 p.

00/86 D. BUYENS, A. DE VOS, The added value of the HR-department : empirical study and development of anintegrated framework, June 2000, 37 p.

00/87 K. CAMPO, E. GIJSBRECHTS, P. NISOL, The impact of stock-outs on whether, how much and what to buy, June2000, 50 p.

00/88 K. CAMPO, E. GIJSBRECHTS, P. NISOL, Towards understanding consumer response to stock-outs, June 2000,40 p. (published in Journal of Retailing, 2000)

00/89 K. DE WULF, G. ODEKERKEN-SCHRÖDER, P. SCHUMACHER, Why it takes two to build succesful buyer-sellerrelationships July 2000, 31 p.

00/90 J. CROMBEZ, R. VANDER VENNET, Exact factor pricing in a European framework, September 2000, 38 p.

00/91 J. CAMERLYNCK, H. OOGHE, Pre-acquisition profile of privately held companies involved in takeovers : anempirical study, October 2000, 34 p.

00/92 K. DENECKER, S. VAN ASSCHE, J. CROMBEZ, R. VANDER VENNET, I. LEMAHIEU, Value-at-risk predictionusing context modeling, November 2000, 24 p. (forthcoming in European Physical Journal B, 2001)

00/93 P. VAN KENHOVE, I. VERMEIR, S. VERNIERS, An empirical investigation of the relationships between ethicalbeliefs, ethical ideology, political preference and need for closure of Dutch-speaking consumers in Belgium,November 2000, 37 p. (forthcoming in Journal of Business Ethics, 2001)

00/94 P. VAN KENHOVE, K. WIJNEN, K. DE WULF, The influence of topic involvement on mail survey responsebehavior, November 2000, 40 p.

00/95 A. BOSMANS, P. VAN KENHOVE, P. VLERICK, H. HENDRICKX, The effect of mood on self-referencing in apersuasion context, November 2000, 26 p. (forthcoming in Advances in Consumer Research, 2001)

00/96 P. EVERAERT, G. BOËR, W. BRUGGEMAN, The Impact of Target Costing on Cost, Quality and DevelopmentTime of New Products: Conflicting Evidence from Lab Experiments, December 2000, 47 p.

00/97 G. EVERAERT, Balanced growth and public capital: An empirical analysis with I(2)-trends in capital stock data,December 2000, 29 p.

00/98 G. EVERAERT, F. HEYLEN, Public capital and labour market performance in Belgium, December 2000, 45 p.

00/99 G. DHAENE, O. SCAILLET, Reversed Score and Likelihood Ratio Tests, December 2000, 16 p.

FACULTEIT ECONOMIE EN BEDRIJFSKUNDE HOVENIERSBERG 24 9000 GENT Tel. : 32 - (0)9 – 264.34.61

Fax. : 32 - (0)9 – 264.35.92

WORKING PAPER SERIES 6

01/100 A. DE VOS, D. BUYENS, Managing the psychological contract of graduate recruits: a challenge for humanresource management, January 2001, 35 p.

01/101 J. CHRISTIAENS, Financial Accounting Reform in Flemish Universities: An Empirical Study of the implementation,February 2001, 22 p.