ELSEVIER International Journal of Forecasting 12 (1996) 1-8
Edi to r i a l
The role and validity of judgment in forecasting
G e o r g e Wr igh t a'*, Michae l J. L a w r e n c e b, F r e d C o l l o p y c
aSchool of Business and Economic Studies, University of Leeds, Leeds LS2 9JT, UK bSchool of Information Systems, University of New South Wales, Sydney 2052, N.S.W., Australia
CWeatherhead School of Management, Case Western Reserve University, 10900 Euclid Avenue, Cleveland, OH 44106-7325, USA
Abstract
All forecasting methods involve judgment but forecasting techniques are often dichotomised as judgmental or statistical. Most forecasting research has focused on the development and testing of statistical techniques. However, in practice, human reasoning and judgment play a primary role. Even when statistical methods are used, results are often adjusted in accord with expert judgment (Bunn and Wright 1991). This editorial introduces the papers included in this special issue of the International Journal of Forecasting and places them within a broader research context. The discussion of this context is structured in three sections: judgmental probability forecasting; judgmental time series forecasting; and interaction of judgment and statistical models.
Subjective probabilities are utilised as a prime input to many management technologies such as decision analysis, cross-impact analysis, expert systems and fault-tree analysis. Much of the psychological research on probability judgment has shown that people's thinking about subjec- tive probability often does not follow the prob- ability laws and, instead, appears to follow heuristic principles that can lead to bias in judgments.
Normative adequacy reflects the degree to which the assessments conform to the axioms of probability. However, probability assessments
* Corresponding author.
can also be examined for their accuracy in the light of subsequent events. Lichtenstein et al. (1982) and McClelland and Bolger (1994) thus delineated a further aspect of a probability assessor's adequacy, 'calibration'. A probability assessor is 'well calibrated', if, over the long run, for all propositions assigned the same probabili- ty, the proportion that are true is equal to the probability assigned. (See Yates, 1990, for a full discussion of calibration and related measures). The great majority of studies of calibration have used 'almanac' questions such as 'which is longer, (a) the Suez Canal or (b) the Panama Canal?' A respondent is typically asked to indi- cate which of the two answers is correct and then indicate how sure he/she is by writing in a probability between 0.5 and 1.0. General knowl-
edge questions have been extensively used by researchers because the answers people gave could be immediately and conveniently evalu- ated by the experimenter (c.f. Wright and Ayton, 1986). Indeed, it was not until the mid 1980s that the focus of calibration studies moved from general knowledge verification tasks to judgmen- tal probability forecasting. Note that calibration is a relative frequency standard of evaluation that can be applied to a set of repetitive forecasts or to a set of forecasts for one-off, unique events. We return to this issue later in this section.
Wright and Ayton (1989) showed that increas- ing personal involvement with a future event tends to invoke a greater feeling of certainty that the event will or will not occur. With personal events, which could happen within a 4 week time horizon, increasing subjective desirability of an event was associated with overconfidence in its likelihood of occurrence. Wright and Ayton (1989) also found that if a forecaster feels that the occurrence of events is under his or her control, this perceived controllability results in increased overconfidence.
The duration of the time period and its immi- nence are two obvious factors which one might expect to have some influence on judgmental probability forecasting. Milburn (1978) investi- gated the effect of imminence and found subjects perceived desirable (non-personal) events as becoming increasingly more likely to occur in each of four successive decades in the future. In contrast, undesirable events were perceived as becoming less likely in each of the four succes- sive decades. Milburn suggested that the availa- bility heuristic (Tversky and Kahneman, 1974) predicts that people will feel that the world may change more and become progressively less like the present in successive decades. Thus it should be harder to imagine what the world will be like. Desirable events increase in probability over time because, Milburn argues, desirability has a greater effect in more ambiguous circumstances. There was no attempt to assess the calibration of these forecasts since this was not feasible due to the large time periods involved.
In an empirical study of the effects of immi- nence, time duration, and subjective desirability
on judgmental probability assessment and sub- sequent calibration, Wright and Ayton (1987) asked forecasters to assess probabilities for 70 non-personal events. Their study found that the effect of increasing the time duration of a fore- cast period from 1 to 2 months had no effect on forecasting response and performance. There- fore, Howell and Burnett's (1978) speculation that shorter duration forecasts will be more prone to unstable ephemeral influences did not hold true for one- and two-month forecasts of impersonal events.
In general, across and within Wright and Ayton's (1987) immediate and short-term fore- casting periods, the subjective desirability of the non-personal events had no effect on their per- ceived likelihood of occurrence. The result con- trasts with Wright and Ayton's (1989) findings of a strong desirability effect found with personal events. However, Wright and Ayton (1992) found that, in medium-term forecasting, the desirability of an event's occurrence will inflate its assessed likelihood which will result in re- duced calibration and increased overconfidence. This result is particularly worrying for the prac- tice of decision analysis where subjective prob- ability assessments for a variety of desired and favored events are routinely elicited by decision analysts from those decision-makers who enlist their help in resolving difficult decisions in the fact of uncertainty. It follows that in medium- term forecasting, ambiguity and lack of infor- mation may result in a stronger interaction between the utility of an outcome and the judged likelihood of its occurrence (cf. Milburn, 1978). This result parallels that of Wright and Ayton (1989) with personal events and contrasts with that of Wright and Ayton (1987) who used non- personal events combined with a weaker ma- nipulation of the imminence dimension.
Additionally, Wright and Ayton (1992) found no support for the hypothesis that calibration is better for the 6-month forecasting period as opposed to 1-month periods. This result can perhaps be explained in terms of the events contained in their questionnaires. Each of the event questions was unique and, arguably, base rate information appropriate to the particular questions posed might not be readily available. It
Editorial 3
follows from this analysis that an analogy with short-run and long-run predictions of events (c.f. Howell and Burnett, 1978; Gigerenzer, 1994), (where a relative frequency 'reframing' of the question posed is possible) is unlikely. If they had, instead, asked for forecasts of weather states or utilised gambling scenarios then predic- tive conditions more likely to evoke relative frequency reframing would have been applied. As Gigerenzer (1994) has shown, if a probability assessor is enabled to frame an assessment as one of relative frequency then validity (ie, cali- bration) improves. Clearly, increasing the time duration of the forecast period, per se, may not directly improve forecasting performance.
In two studies, Wright and Ayton (1988, 1989) have found evidence for consistent individual differences in forecasting response and perform- ance across a range of situations differing in desirability, imminence, time duration and for personal and non-personal events. Ayton and Wright (1987) argued that those people whose probability assessments are more coherent (i..e conform more to the probability laws) should show better forecasting performance than those people whose probability assessments are less coherent. The argument is based on the analogy with reliability and validity measurement in the psychometric approach to questionnaire design and analysis, with coherence being analogous to questionnaire reliability (internal consistency) and calibration being analogous to questionnaire validity (predictive validity). In questionnaire design, adequate reliability is a necessary, though not sufficient condition for adequate validity. Wright et al. (1994) explored the rela- tionship between coherence and calibration and found the effect moderated by the subjectively- rated expertise of the probability assessors. In general, those forecasters who professed greater substantive expertise about their forecasts were better calibrated.
Onkal and Muradoglu (this issue) examine portfolio managers' probability forecasts of stock price movements over a one-week time interval. Using expert, semi expert and novice forecasters they found that the expert portfolio managers showed superior performance compared to the other groups, especially when the forecast assess-
ment was made using a fine-grained multiple interval scale rather than a dichotomous increase or stay-the-same/decrease response format. Ad- ditionally, they found that experts performed better than semi-experts for shorter forecast horizons but that the difference was reversed as the forecast horizon was extended.
Wilkie and Pollock (this issue) examine sub- jective probability forecasting in the currency markets. They utilise and extend the 'covariance decomposition' approach developed by Yates (1982) and generate a general framework for examining the quality of probability judgment in currency forecasting contexts. A covariance de- composition of subjective probability forecasts includes calibration as one of several measures of forecasting adequacy.
Yates et al. (this issue) investigate how prob- abilistic forecasts are evaluated by experimental subjects who are untrained in probability fore- cast evaluation techniques such as scoring rules, calibration, and covariance graphs. Their results show that forecast 'consumers' prefer forecasts that are (1) extreme, since through this fore- caster competence may be inferred, and (2) evidence a documented process by which the probability assessments were made. Interesting- ly, 50% or 'don't know' forecasts tend to be viewed with disdain since forecaster ignorance is apparently inferred. However, formal evaluation methods to characterise the overall accuracy of probabilistic forecasts place no emphasis on the adequacy of the judgmental process or on the distribution of forecast responses, per se.
Abramson et al. (this issue) provide a detailed case description of the process of building an inherently probabilistic forecasting system to forecast severe weather patterns. They identify several important issues associated with model- ing situations in which a large number of prob- abilities interact. Establishing the system's boundaries, applying scenarios to simplify a complex domain, and providing support for the estimation of coherent probability assessments are three areas in which the case suggests that research might profitably be done.
To date, the focus of the research on judg- mental probability forecasting has been on com- paring the quality of numerical judgments of the
4 Editorial
likelihood of forecast events with the outcomes that, in fact, occur. A key finding has been overconfidence, where the probability assigned to a set of events is more than the subsequent hit-rate. However, our knowledge of the nature of the reasoning process which underpins the numerically assessed response is less well-de- veloped. Benson et al. (1996), Curley et al. (1995) have argued that reasoning has been rarely studied in the context of probability as- sessment but, arguably, probability assessment can be improved by a focus on improving reason- ing rather than attempting to improve tbe output response by, for example, identifying whether the occurrence of a forecast event is highly desirable to the forecaster and thus, mechanical- ly, adjusting downwards the forecaster's assess- ment of its likelihood of occurrence.
Rowe and Wright (this issue) focus on the role of reasoning in judgmental forecasting. They analyse how judgments at the individual level are influenced by the judgments and rationales of other individuals in nominal, Delphi groupings. Rowe and Wright investigate the influence of such factors on individual opinion change and subsequent judgment accuracy.
2. Judgmental time series forecasting
The inputs to estimating a time series forecast are generally the historical data together with contextual data. In the case of sales forecasts, the contextual data may be promotion plans, estimates of competitor reactions and micro- economic expectations. Because contextual data is frequently soft and not amenable to quantita- tive manipulation, forecast estimation is usually a combination of part quantitative and part subjective or judgmental. Even the most die- hard quantitative forecaster may resort to judg- mental adjustment of the time series history to remove extreme events caused by strikes or special promotional events. And even the most die-hard judgmental forecaster may utilise quan- titative methods to process the time series his- tory before estimating a forecast. However, we tend to dichotomise forecasting as quantitative
or judgmental based principally on how the time series data is extrapolated.
Surveys of forecasting practice in Australia, USA and UK indicate the continuing high use of judgmental and opinion based methods in pre- ference to quantitative methods (e.g. Sparkes and McHugh, 1984; Dalrymple, 1987; Tarranto, 1989; Sanders and Manrodt, 1994). These studies of business (mostly sales) forecasting practice reveal that only around 10% of the firms sur- veyed use quantitatively based forecasting tech- niques and that the number of firms who have tried and subsequently abandoned these tech- niques is about double the number currently using them. This situation cannot be blamed on limitations of current forecasting technology- there has been an enthusiastic response by system developers and there is a wide range of inexpensive and user-friendly PC Windows and main-frame-based software to choose from.
These systems benefit from the great volume of forecasting research conducted over the last 25 years. As might be expected, this lack of use has been a cause of concern to system develop- ers, researchers and management educators who believe that valuable technology is remaining under-utilised (Makridakis, 1988; Armstrong, 1994).
Early expectations (e.g. Hogarth and Mak- ridakis, 1981) were that the accuracy of judg- mental extrapolation was inferior to quantitative extrapolation. (By extrapolation here we indi- cate forecasting where no contextual data is used.) However, the work of Lawrence (1983) and a larger study by Lawrence et al. (1985) using the M-Competition database of 111 time series and their quantitative forecasts (Mak- ridakis et al., 1982), showed that judgmental extrapolation was about as accurate as the best quantitative extrapolation.
From rather uncertain beginnings, motivated by the practical use of judgmental forecasting and its comparative accuracy, the last ten years have seen an increasing output of research in this field. Webby and O'Connor (this issue) present a review of this literature. We adopt a slightly different framework and broadly group this work into the following categories, each of which is
Editorial 5
briefly discussed; forecast accuracy studies; un- derstanding and improving judgmental forecast- ing; and; accuracy of judgmental prediction intervals.
issues to do with how contextual information is utilised by the forecaster and give practical guidelines for when judgmental interventions in statistical models are likely to be beneficial.
2.1. Forecast accuracy studies 2.2. Accuracy of judgmental prediction intervals
These studies can be classed broadly into real life data and artificial data, and whether or not contextual data is present. Most of the studies have not included contextual data, perhaps re- flecting the laboratory nature of much of the work.
When no contextual data is present, we are dealing with extrapolation--both quantitative and judgmental forecasting are restricted to the same time series data set. Comparison is thus on a 'level field'. An initial general observation is that when artificial data is used, accuracy favors quantitative forecasting (e.g. Adam and Ebert, 1973; Lawrence and O'Connor, 1992). On the other hand, when real life data is used the results are about the same, particularly for sales type data (Lawrence et al., 1985; Makridakis et al., 1993). Clearly, the instability and non-stationari- ty of real life data favors the subjective and penalises the quantitative method which relies on stationarity and the assumption of constancy.
Relatively few studies have been carried out to compare forecasts made under real life condi- tions when contextual data was used by the judgmental forecaster. Under these conditions, the quantitative forecasts are only informed by the time series. Thus studies in this category are, in effect, measuring the human ability to use contextual data. Edmundson et al. (1988) and Fildes (1991) showed that judgmental forecasts using contextual data were significantly more accurate than quantitative forecasts. However, Lawrence et al. (1995) and Sanders and Ritzman (1992) found mixed evidence. It thus appears that mixed results are being achieved in real life sales or sales-type forecasting. In forecasting company earnings, the picture is more on the side of judgmental forecasting being more accur- ate than extrapolation (e.g. Hopwood and McK- eown, 1990; Brown et al., 1987; Armstrong, 1983). Goodwin and Wright (1993, 1994) discuss
Prediction intervals are frequently estimated and supplied with forecasts (Dalrymple, 1987) as they constitute an important element in making a decision using that forecast. When the forecast is made judgmentally, the prediction interval will generally also be estimated judgmentally. Evi- dence on the accuracy and calibration of judg- mental prediction intervals is not very encourag- ing. Lawrence and Makridakis (1989), O'Connor and Lawrence (1989) and Lawrence and O'Con- nor (1993) showed that the accuracy and cali- bration of judgmental prediction intervals are strongly influenced by seasonality, trend, ran- domness and the scale of a graphical plot used to present the time series to the subject.
2.3. Understanding and improving judgmental forecasting
The studies mentioned above all accept judg- mental forecasting as a holistic 'eyeball' estima- tion of a time series. Another line of research has sought to improve the process of judgmental forecasting either through computer support or by attempting to remove its biases by structuring its process. Edmundson (1990) built a decision support tool called ORAFFECr which decomposed the forecasting task into the classical components of trend, seasonal and randomness. The tool supported judgmental estimation of trend and seasonal and allowed the direct entry of the impact of the contextual data to the deseasonal- ised forecast. Tests of GRAFFECT showed it to improve the accuracy of both novice and expert forecasters when compared to either unaided estimation or deseasonalised single exponential smoothing.
Judgmental forecasting has been shown to be subject to biases arising from underestimation of trend (Bolger and Harvey, 1993), to give exces- sive weight to the most recent segment slope
6 Editorial
(Lawrence and O'Connor, 1992), and to be prone to incorporating the noise component into the forecast in order to make the forecasts look like the data series (Harvey, 1995). However, judgmental forecasts do not conform to the usual anchor and adjust bias whereby the adjustment from the anchor point is too little--in fact the adjustment tends to be generally too much, probably reflecting the 'goodness' of the last observation (the naive forecast) as the anchor point (Lawrence and O'Connor, 1995). In this issue, Harvey and Bolger address the long stand- ing question of whether the data should be displayed in a graph or a table for estimating a judgmental forecast. This is an important issue which purveyors of graphically based software seem to think is very obvious.
Harvey and Bolger show that the answer depends on what type of series you are attempt- ing to forecast. If you want to forecast a trended series then a graphical plot appears to be better but if there is no trend in the series a tabular display is superior. The reasons for these findings appear in part to be due to the tendency for subjects using a graph to overforecast and be less consistent than for tabular subjects. As most business judgmentally based sales forecasting tends to be performed using tabular displays this finding should be of practical interest for those forecasting trended series.
3. Interaction of judgment and statistical models
Interest in the interaction of judgment and statistical models seems to be growing among forecasting researchers. There appears to be several reasons for this. Foremost is a desire to incorporate more contextual knowledge into forecast~. Over a decade ago, in her response to the M-Competition, Lopes (1983) suggested that 'extrapolative methods would be strengthened by making provisions for causal or explanatory information to be used when such is available.' The robustness of the combining strategy has encouraged many researchers to consider com- bining forecasts that have been made judgmen- tally (and which presumably incorporate causal
or explanatory information) with forecasts from statistical models. The success of decision sup- port systems has produced environments in which mathematical models and judgment can be easily integrated.
Firms using quantitative forecasting techniques typically resort to adjusting the statistical ex- trapolation when contextual data is believed to invalidate the assumption of constancy (e.g. a broken leg cue). Studies attempting to measure the benefit from such forecast adjustments in real world settings have generally found that it reduces errors over the base quantitative fore- casts (e.g. Mathews and Diamantopoulos, 1990). Bunn and Wright (1991) have raised the broader question of just how to structure the combina- tion of judgmental knowledge and quantitative forecasts. One response by Armstrong and Col- lopy (1993) is to use the notion of causal forces.
The current issue contains three papers that directly address problems encountered when using judgment to adjust forecasts from statisti- cal models. Lim and O'Connor (this issue) investigate the way in which people adjust statistical forecasts in the light of contextual information. People are able to utilise causal information but are only partially sensitive to variations in its predictive validity.
In Belton and Goodwin (this issue), attempts to use the analytic hierarchy process to make judgmental adjustments are critiqued. The au- thors express concern that decision makers may not understand the questions they are asked about their beliefs. More importantly, they note that a method designed to make relative com- parisons is being applied to determine absolute adjustments. Suggestions for modifications that might circumvent these dangers are presented.
Bunn and Salo (this issue) deal with a more general problem in judgmental adjustments of forecasts from statistical models, double-count- ing bias. They describe a screening procedure that can be useful in determining if judgmental adjustments are necessary, since often the vari- able of interest is already represented implicitly in the model (due to collinearity with other variables that have been used in estimating the model). After presenting a theoretical argument for the use of model consistent expectations in
Editorial 7
making decisions about such judgmental adjust- ments, they illustrate using a capital cost fore- casting problem faced by a major oil company.
These papers illustrate important problems faced by those wishing to make judgmental adjustments, and serve as warnings that a care- less approach carries with it risks. Together, the papers offer elements of a protocol that might be employed in more effectively using an adjust- ment strategy. It is perhaps too much to expect, though, that these protocols will be widely em- ployed unless they are embedded in expert and decision support systems, and thereby made available to forecasters as they are needed.
Given the clear preference of many practition- ers to use judgmentally based forecast methods, if the academic forecasting researchers are to usefully interact with and improve practice then it seems that improving judgmental forecasting is a high priority. Some of the issues that we believe should be studied by academic research- ers include the design of computer-based support to reduce the prevalence of potential biases- either through (re)structuring the forecasting task or through training the forecasters. Feed- back to judgment forecasters on the degree of coherence in their forecasts and on the sub- sequent validity of their forecasting performance should improve the accuracy of forecasting judg- ment (c.f. Bolger and Wright, 1994) but this area is also, as yet, underexplored. The combination of judgment and statistical methods is a par- ticularly promising area for researchers inter- ested in improving practice. Attention should be directed at establishing the specific conditions under which statistical methods, judgment alone, and combinations of the two are likely to be most useful. Usefulness should be taken to refer to more than improved accuracy. It should also include systematic consideration of such factors as the costs of producing the forecasts, the availability of calibrated confidence intervals (when such would be useful), and the perceived validity of the resulting forecasts.
References
Adam, E.E. and R.J. Ebert, 1973, A comparison of human and statistical forecasting, AIIE Transactions 8, 120-127.
Armstrong, J.S,, 1983, Relative accuracy of judgmental and extrapolative methods in forecasting annual earnings, Journal of Forecasting 2, 437-447.
Armstrong, J.S., 1994, Review of Sanders and Manrodt, International Journal of Forecasting 10, 471-472.
Armstrong, J.S. and F. Collopy, 1993, Causal forces: Struc- turing knowledge for time-series extrapolation, Journal of Forecasting 12, 103-115.
Ayton, P. and G. Wright, 1987, Assessing and improving judgmental probability forecasts, Omega, International Journal of Management Science 15, 191-196.
Benson, P.G., Curley, S.P. and G.F. Smith, 1996, Belief assessment: An underdeveloped phase of probability elici- tation, Management Science, in press.
Bolger, F., and N. Harvey, 1993, Context-sensitive heuristics in statistical reasoning, Quarterly Journal of Experimental Psychology 46A, 779-811.
Bolger, F. and G. Wright, 1994, Assessing the quality of expert judgment: Issues and analysis, Decision Support Systems 11, 1-24.
Brown, L.D., Hagerman, R.L., Griffin, P.A. and M.E. Zmijewski, 1987, Security analyst superiority relative to univariate time series models in forecasting quarterly earnings, Journal of Accounting and Economics 9, 61-87.
Bunn, D. and G. Wright, 1991, Interaction of judgmental and statistical forecasting methods: Issues and analysis, Man- agement Science 37, 501-518.
Curley, S.P., Browne, G.J., Smith, G.F. and P.G. Benson, 1995, Arguments in the practical reasoning underlying constructed probability responses, Journal of Behavioral Decision Making 8, 1-20.
Dalrymple, D.J., 1987, Sales forecasting practices: Results from a US survey, International Journal of Forecasting 3, 379-391.
Edmundson, R.H., Lawrence, M.J. and M.J. O'Connor, 1988, The use of non-time series data in sales forecasting: a case study, International Journal of Forecasting 7, 201- 212.
Edmundson, R., 1990, Decomposition: a strategy for judg- mental forecasting, Journal of Forecasting 4, 305-314.
Fildes, R., 1991, Efficient use of information in the formation of subjective industry forecasts, Journal of Forecasting 10, 597-617.
Gigerenzer, G., 1994, Why the distinction between single- event probabilities and frequencies is important for psy- chology and vice versa. In G. Wright and P. Ayton, eds., Subjective Probability, (Wiley, Chichester).
Goodwin, P. and G. Wright, 1993, Improving judgmental time series forecasting: a review of the guidance provided by research, International Journal of Forecasting 9, 147- 161.
Goodwin, P. and G. Wright, 1994, Heuristics, biases and improvement, strategies in judgmental time series forecast- ing, Omega, International Journal of Management Science 22, 553-568.
Harvey, N., 1995, Why are judgments less consistent in less predictable task situations?, Organisational Behavior and Human Decision Processes 63, 247-263,
8 Editorial
Hogarth, R.M. and S. Makridakis, 1981, Forecasting and Planning: An Evaluation, Management Science 27, 115- 137.
Hopwood, W.S. and J.C. McKeown, 1990, Evidence on surrogates for earnings expectations within a capital mar- ket context, Journal of Accounting, Auditing and Finance 5, 339-368.
Howell, W.C. and S.A. Burnett, 1978, Uncertainty measure- ment: a cognitive taxonomy, Organisational Behavior and Human Performance 30, 45-68.
Lawrence, M., 1983, An exploration of some practical issues in the use of quantitative forecasting models, Journal of Forecasting 1, 169-179.
Lawrence, M.J., Edmundson, R.H. and M.J. O'Connor, 1985, An examination of the accuracy of judgmental extrapolation of time series, International Journal of Fore- casting 1, 25-35.
Lawrence, M.J., Edmundson, R.H. and M.J. O'Connor, 1995, A Field Study of Sales Forecasting: Its Accuracy, Bias and Efficiency, Working Paper, (School of Infor- mation Systems, Univ of New South Wales, Australia).
Lawrence, M.J. and S. Makridakis, 1989, Factors affecting judgmental forecasts and confidence intervals, Organisa- tional Behavior and Human Decision Processes 43, 172- 187.
Lawrence, M.J. and M.J. O'Connor, 1992, Exploring Judg- mental Forecasting, International Journal of Forecasting 8, 15-26.
Lawrence, M.J. and M.J. O'Connor, 1993, Scale, Random- ness and the Calibration of judgmental confidence inter- vals, Organisational Behavior and Human Decision Pro- cesses 56, 441-458.
Lawrence, M.J. and M.J. O'Connor, 1995, The anchoring and adjustment heuristic in time series forecasting, Journal of Forecasting 14, 443-451.
Lichtenstein, S., Fischhoff, B. and L.D. Phillips, 1982, Calibration of probabilities: the state of the art to 1980.In D. Kahneman, E Slovic and A. Tversky, eds., Judgment Under Certainty: Heuristics and Biases, (Cambridge Uni- versity Press, Cambridge).
Lopes, L., 1983, Pattern, Pattern--Who's Got the Pattern? Journal of Forecasting 2, 269-272.
Makridakis, S., 1988, Metaforecasting, International Journal of Forecasting 4, 467-491.
Makridakis, S., Anderson, A., Carbone, R., Fildes, R., Hibon, M., Lewandowski, R., Newton, J., Parzen, E. and R. Winkler, 1982, The accuracy of extrapolation (time series) methods: results of a forecasting competition, Journal of Forecasting 1, 111-153.
Makridakis, S., Chatfield, C., Hibon, M., Lawrence, M., Mills, T., Ord, K. and L. Simmons, 1993, The M2-Compe- tition: A real time judgmentally based forecasting study, International Journal of Forecasting 9, 5-22.
Mathews, B.P. and A. Diamantopoulos, 1990, Judgmental revision of sales forecasts: Effectiveness of forecast selec- tion, Journal of Forecasting 9, 407-415.
McClelland, A. and F. Bolger, 1994, The Calibration of Subjective Probabilities: Theories and Models 1980-94. In G. Wright and P. Ayton, eds., Subjective Probability, (Wiley, Chichester).
Milburn, M.A., 1978, Sources of bias in the prediction of future events, Organisational Behavior and Human Per- formance 21, 17-26.
O'Connor, M.J. and M.J. Lawrence, 1989, An examination of the accuracy of judgmental confidence intervals in time series forecasting, Journal of Forecasting 8, 141-156.
Sanders, N. and L. Ritzman, 1992, The need for contextual and technical knowledge in judgmental forecasting, Jour- nal of Behavioral Decision Making 5, 39-52.
Sanders, N. and K. Manrodt, 1994, Forecasting practices in US corporations: survey results, Interfaces 24, 92-100.
Sparkes, J.R. and A.K. McHugh, 1984, Awareness and use of forecasting techniques in British industry, Journal of Forecasting 3, 37-42.
Tarranto, M., 1989, Sales forecasting practice in Australia, unpublished B Corn honours thesis, (University of NSW, Kensington, NSW),
Tversky, A. and D. Kahneman, 1974, Judgment under certainty: Heuristics and biases, Science 185, 1124-1131.
Wright, G. and P. Ayton, 1986, Subjective confidence in forecasts: A response to Fischhoff and MacGregor, Jour- nal of Forecasting 5, 117-123.
Wright, G. and E Ayton, 1987, Tasks influences on judg- mental forecasting, Scandinavian Journal of Psychology 28, 115-127.
Wright, G. and E Ayton, 1988, Immediate and short-term judgmental forecasting: personologism, situationism, or interactionism? Personality and Individual Differences 9, 109-120.
Wright, G. and P. Ayton, 1989, Judgmental probability forecasts for personal and impersonal events, International Journal of Forecasting 5, 117-126.
Wright, G. and P. Ayton, 1992, Judgmental probability forecasting in the immediate and medium term, Organisa- tional Behavior and Human Decision Processes 51, 344- 363.
Wright, G., Rowe, Bolger, F. and J. Gammack, 1994, Coherence, calibration and expertise in judgmental prob- ability forecasting, Organisational Behavior and Human Decision Processes 57, 1-25.
Yates, J.F., 1982, External correspondence: Decompositions of the mean probability score, Organizational Behavior and Human Performance 30, 132-156.
