Bernoulli Bayes Laplace 1713 1763 1813

Anniversary Volume

Proceedings of an International Research Seminar Statistical Laboratory

University of California, Berkeley 1963

Edited by Jerzy Neyman and Lucien M. Le Cam

Springer-Verlag Berlin Heidelberg GmbH 1965

All rights, especially that of translation into foreign languages, reserved. It is also forbidden to reproduce this book, either whole or in part, by photomechanical means (photostat, microfilm andfor microcard) or by other procedure without written

permission from Springer-Verlag.

© Springer-Verlag Berlin Heidelberg 1965 Originally published by Springer-Verlag Berlin Heidelberg New York in 1965

ISBN 978-3-642-49467-3 ISBN 978-3-642-49750-6 (eBook) DOI 10.1007/978-3-642-49750-6

Library of Congress Catalog Card Number 65-26 236

Title No. 1286

Foreword The present volume represents the Proceedings of an International

Research Seminar organized in 1963 by the Statistical Laboratory, Uni­versity of California, Berkeley, on the occasion of a remarkable triple anniversary:

the 250th anniversary of jACOB BERNOULLI's "Ars Conjectandi", the 200th anniversary of THOMAS BAYES' "Essay towards solving a

problem in doctrine of chance", and the !50th anniversary of the PIERRE-SIMON DE LAPLACE's "Essai

philosophique sur les probabilites". Financial assistance of the National Science Foundation, without

which the Seminar could not have been held, is gratefully acknowledged. The publication of Ars Conjectandi, in 1713, was a milestone in the

history of probability theory. Here, for the first time, appeared a careful description of the now well-known combinatorial methods which give solutions of many problems on simple games of chance. Also, Ars Conjectandi contains the Bernoulli numbers, theorems relating to the duration of games, and to the ruin of gamblers and, above all, the state­ment and proof of the famous Bernoulli weak law of large numbers.

Even though the original Latin edition of Ars Conjectandi was followed by several in modern languages, currently the book is not easily accessible. Apparently the last re-publication, in German, occurred in 1899, in two issues, No. 107 and No. 108, of the series "Ostwald's Klassi­ker der exakten Wissenschaften", Wilhelm Engelman, Leipzig. The two books are difficult to locate.

In 1763, exactly 50 years after Ars Conjectandi, THOMAS BAYES' "Essay" was posthumously published by Richard Price (Philosophical Transactions, Royal Society, London, Vol. 53, 1763, pp. 376-398). This paper has been the focus of what may be the most heated controversy in the history of probability and statistics, a controversy that extends to the present time, The contents of Bayes' paper are limited and mathematic­ally unsophisticated, and the most surprising thing about the paper is that it could have become the center of frequently bitter and prolonged debate.

Bayes' ,Essay" is readily accessible. Sometime in the 1930's it was photographically republished by the Graduate School of the U.S. De­partment of Agriculture, Washington, D.C., with commentaries by W. EDwARDS DEMING and EDWARD C. MoLINA. More recently it was again re-published, with commentaries by G. A. BARNARD, in Biometrika, Vol. 45 (1958).

IV Foreword

In 1812, 49 years after the appearance of Bayes' paper, the French Academy published the memorable book "Theorie analytique des pro­babilites" by PIERRE-SIMON DE LAPLACE. In spite of the then developing Napoleon's debacle in Russia, the book must have sold well, presumably not only in France, because the second edition appeared in 1814, only two years later. In addition to the original text of almost 500 pages, this second edition contains several supplements and a 153 page "I ntro­duction". This "Introduction", then, must have been written in 1813, 150 years before the Berkeley Seminar of 1963. It appeared also as a separate publication, under the title "Essai philosophique sur les pro­babilites".

"Theorie analytique", including the Introduction, has again been republished in 1820 and several times thereafter and is currently acces­sible in many university libraries. An English version of the "Essai philosophique" was issued in 1951 by Dover Publications.

The interest that a contemporary reader may find in the three famous publications must be conditioned by two factors: the character of contents and the time interval dividing us from the epoch when the given work was completed. These two factors combine heavily to favor LAPLACE. In fact, we found "Theorie analytique" not only readable, but highly interesting and thoroughly enjoyable, both because of its contents and because of the elegance of LAPLACE's style. Regretfully, this elegance is easily lost in translations.

"Essai philosophique" is described by LAPLACE as an extended text of a lecture he delivered in the Ecoles Normales in 1795, and con­tains no mathematics. Essentially, it may be considered as a summary, of LAPLACE's knowledge in the various domains of science and of his thinking over the period between the French Revolution and the year of disaster marking the decline of the Napoleonic era. This by itself makes "Essai philosophique" very interesting.

The leading idea of the book is that each particular phenomenon in Nature, including social and political developments, is governed by forces of two distinct kinds, the permanent forces and the accidental forces. In each particular phenomenon, the effect of accidental forces may appear stronger than that of permanent forces, with the result that such phenomena become appropriate subjects for probabilistic studies. On the other hand, in a long series of similar occurrences, the accidental forces average out and the permanent forces prevail. This is considered by LAPLACE as a consequence of Bernoulli's law of large numbers and LAPLACE is emphatic in praising BERNOULLI. Considerations of the above kind are, of course, quite usual with reference to lotteries, games of dice, insurance, and so on. However, LAPLACE's musings go much farther. Here is an illustration.

Foreword v

"This theorem (the weak law of large numbers) implies also that, in the long run the action of regular and constant forces must prevail upon that of irregular forces. It is this circumstance that makes the earnings of lotteries as certain as those of agriculture: the chances reserved for the lottery insure its gains in the total of a large number of games. Similarly, since numerous favorable chances are tied with the eternal principles of reason, justice and humanity, the principles that are the foundation of societies and their mainstays, there is a great advantage in adhering to these principles and serious inconveniences in departing from them. Both history and personal experiences support this theoretical result. Consider the benefits to the nations from institutions based on reason and on the natural rights of man, the nations who knew how to establish such in­stitutions and how to maintain them. Consider also the advantages that good faith earns governments whose policies are based on good faith, and how richly these governments are repaid for the sacrifices incurred in scrupulous observance of their commitments. What immense internal credit! What authority abroad! Consider, on the contrary, the abyss of miseries into which the peoples are frequently thrown by the ambitions and treachery of their rulers. Whenever a great power, intoxicated by lust for conquests, aspires to world domination, the spirit of independence among the menaced nations leads to a coalition, to which the aggressor power almost invariably succumbs. Similarly, the natural boundaries of a State, acting as constant causes, must eventually prevail over the variable causes that alternatively expand or compress the given State. Thus, it is important for stability, as well as for the happiness of empires, not to extend them beyond the limits into which they are repeatedly thrown by the constant causes, just as ocean waves whipped up by violent tempest fall back into their basin due to gravitation. This is an­other example of a probability theorem being confirmed by disastrous experiences.''

Clarity of the concept of probability and of its relation to physical phenomena was reached early in the 20th century mainly through the works of KOLMOGOROV, on the one hand, and of VON MISES, on the other. Thus, LAPLACE's interpretation of probability is far from consistent and unambiguous. Many of his writings indicate that, for him, probability is a measure of confidence or diffidence, independent of any frequency connotations. If there is no reason to believe that one of the contem­plated events is more likely to occur than any other, then, for LA­PLACE, these events are equiprobable. Here then, the intensity of ex­pectation appears as the decisive moment in assigning probabilities. On the other hand, in many other passages, the decisive role is given to frequencies. For example, in discussing the familiar incident with CHEVALIER DE M:ER.E, LAPLACE appears to consider that the disagree-

VI Foreword

ment between DE M:ERJ§:'s experiments with dice andnEMERE'ssolution of the corresponding probability problem merely confirms the fact, estab­lished by PASCAL and FERMAT, that this solution is wrong. Also, a very substantial section of the "Essai philosophique" is given to "illusions" leading to mistakes in assigning probabilities to events. Here, then, pro­bability appears as something independent of subjective emotions of particular individuals. Incidentally, this section on "illusions" includes LAPLACE's discussion of physiology of the nervous system and of the brain, for which LAPLACE proposes the summary term "Psychology". We were unable to verify whether this now commonly adopted term was actually introduced by LAPLACE.

Frequency interpretation of probability is also apparent in LA­PLACE's studies of a number of applied problems. These studies, described in both the "Essai philosophique" and in the "Thiorie analytique" proper, and also in several sections of "Mecanique celeste", are very much in the spirit of the present day applications of probability and statistics to the various domains of science and we found them very interesting and stimulating. In the 19th and in the early years of the present century, when LAPLACE's writings were read more frequently than they now are, these particular studies exercised a very considerable influence both on theoretical and on applied research. Undoubtedly LAPLACE's discussion of the sex ratio, customarily indicating a prevalence of male births, influenced LEXIS and later BoRTKIEWICZ. Also, LAPLACE's seve­ral studies of comets, indicating that their origin must be different from that of planets, influenced CHARLIER who considerably extended some of them. The same applies to the sections of "Theorie analytique" dealing with the central limit theorem. This book is directly quoted by CHARLIER in his work on asymptotic expansions of probability densities and by HARALD CRAMER. In a sense, the particular sections may be considered as precursors of the entirely novel subdiscipline on "probabilities of large deviations". The element that attracts us most in the "Theorie analytique" is the close tie of the theory with the problems of science: it is the latter that originate non-trivial theoretical problems, the solutions of which generate further theoretical developments.

In general, even though contemporary probabilists and statisticians have gone far beyond LAPLACE in many directions, so that particular subjects treated in "Thiorie analytique" and now are occasionally difficult to identify, we believe that the book is very much worth reading.

A substantial part of the work is devoted to the theory of generating functions. LAPLACE claims to have inherited this from LAGRANGE and LEIBNITZ. However, he proceeds to use (and abuse) the method on various difference, differential and integral operators with an enthusiasm which reappears only much later in the works of Heaviside. One finds in the

Foreword VII

book a substantial use of the method of characteristic functions, also called Laplace transforms or Fourier transforms. This method, also used by LAGRANGE and CAUCHY, presumably independently, was finally revived in the early 20th century by PAUL LEVY, with great success.

The part of the "Theory analytique" relating to "fonctions de tres grands nombres" gave birth to the method of steepest descent and to some understanding of asymptotic expansions. LAPLACE's proof of the central limit theorem cannot be considered rigorous, but it is almost rigorizable, as was finally sho""'Il by LrAPOUNOFF in 1901. A somewhat related result of LAPLACE concerns the behavior of the a posteriori distribution of a parameter given a large number of observations. Although occasionally LAPLACE used an argument of "fiducial" nature, already introduced by J. BERNOULLI and declared objectionable by LEIBNITZ, LAPLACE's treatment of the a posteriori distribution seems basically sound. He anticipated by about a century the proofs of S. BERNSTEIN and VON MrsEs to the effect that, under certain conditions, the a posteriori distribution tends to a normal limit. A pleasant detail here is a careful distinction made by LAPLACE between expected errors computed under the assumption that the observations are random variables and expected errors computed a posteriori assuming the observations fixed.

"Essai philosophique" ends with a historical note covering the period from PASCAL and FERMAT. Here LAPLACE points out the achievements of his several predecessors, including jACOB BERNOULLI (weak law of large numbers), DE MorvRE (central limit theorem) and BAYES. Also, the note mentions the then recent developments regarding the method of least squares. The same subject is again discussed in another historical note in the "Theorie analytique". It is with reference to least squares that LAPLACE conceived the fruitful ideas which, after being for­gotten for a number of years, now serve as foundations of modern statistical theory: the idea that every statistical procedure is a game of chance played with Nature, the idea of a loss function and of risk, and the idea that risk may be used as the basis for defining optimality of the statistical method concerned. LAPLACE's thinking was directed towards the problem of estimation and the loss function he adopted is the absolute value of the error of the estimate. GAuss was quick in recognizing the fruitfulness of these ideas in general, but adopted a more convenient loss function, namely the square of the error.

The details of the discussion conducted a century and a half ago, as well as the then prevailing styles of recognition of priority, are interest­ing and we feel compelled to introduce more quotations, from both LAPLACE and GAuss, as follows.

"In order to avoid this groping in the dark, Mr. LEGENDRE conceived the simple idea of considering the sum of squares of observational

VIII Foreword

errors and of minimizing it. This provides directly the same number of final equations as there are unknowns. This leamed mathematician is the first to publish the method. However, in fairness to Mr. Gauss, it must be ob­served that, several years before this publication, he had the same idea, that he himself used it customarily and that he communicated it to several astronomers .... Undoubtedly, the search for the most advantageous proce­dure (i. e. the procedure minimizing risk) for deriving the final equations is one of the most useful problems in the theory of probability. Its importance for physics and astronomy brought me to study it." (" TMorie analytique", 1820, p. 353).

"The estimation of a magnitude using an observation subject to a larger or to a smaller error can be compared, not inappropriately, to a game of chance in which one can only lose (and never win), and in which each possible error corresponds to a loss. The risk involved in such a game is measured by the probable loss, that is, by the sum of products of particular possible losses by their probabilities. However, what specific loss should one ascribe to any given error is by no means clear by itself. In fact, the determination of this loss depends, at least in part, on our evaluation .... Among the infinite variety of such functions, the one that is the simplest seems to have the advantage and this, unquestionably, is the square. Thus follows the principle just formulated.

LAPLACE treated the problem in a similar fashion, but he choose the absolute value of the error as the measure of loss. However, unless we are mistaken, this choice is surely not less arbitrary than ours." (CARL FRIEDRICH GAuss, "Abhandlungen zur Methode der kleinsten Quadrate", Berlin, 1887, p. 6).

The end paragraph of the "Essai philosophique" begins with the statement: "Essentially, the theory of probability is nothing but good common sense reduced to mathematics. It provides an exact appreciation of what sound minds feel with a kind of instinct, frequently without being able to account for it." The history of least squares, as it emerges from the above quotations, is an illustration of this statement. First came the manipulative procedure of the method that two "esprits justes", first GAuss and then LEGENDRE, advanced on purely intuitive grounds. Next came the efforts at a probabilistic justification of the procedure. Here the priority regarding the basic ideas seems to belong to LAPLACE who, however, was unlucky in the choice of his loss function. The last steps towards the present day foundations of the least squares method, beginning with the square error as the loss function, and culminating with the proof of the theorem about the minimum variance property of least squares estimates among all linear unbiased estimates, are due to GAuss.

The difference between the GAuss and the LAPLACE treatments of

Foreword IX

optimality of the least square solutions is that, in the article quoted, GAuss considers the estimated parameters as unknown constants and minimizes the expected loss with regard to the random variation of the observations. On the contrary, in LAPLACE's treatment it is the parameters that are random variables with some a priori distribution. The method of proof of optimality used here was revived only very recently. Currently it is standard in the asymptotic decision theory.

Before concluding, we wish to express our hearty thanks to all the colleagues who consented to take part in the 1963 International Research Seminar, and to the University of California for providing the necessary facilities. Also we reiterate our expression of gratefulness to the National Science Foundation for the necessary financial help. Finally, cordial thanks are due to Dr. HEINZ GoTZE of Springer-Verlag for his interest in the Seminar and to Springer-Verlag itself for its customary excellent publication of these Proceedings.

LucmN LECAM jERZY NEYMAN

Contents

BARToszv:NsKI, R., J. Los, and l\1. WvcEcH-Los: Contribution to the theory of epidemics . . . . . . . . . . . . . . . . . . . . • . . . . . . . . 1

BLANC-LAPIERRE, A., P. DuMONTET, and B. PICINBONO: Study of some statistic-al models introduced by problems of physics . . . . . . . . . . . . . . 9

BLANC-LAPIERRE, A., and P. FAURE : Stationary and isotropic random functions 17

CoGBURN, R.: On the estimation of a multivariate location parameter with squared error loss . . . . . . . . 24

DAVID, F. N.: Some notes on LAPLACE 30

HAJEK, J. : Extension of the Kolmogorov-Smirnov test to regression alternatives 45

HAMMERSLEY, J. M., and D. J. A. WELsH: First-passage percolation, subad­ditive processes, stochastic networks, and generalized renewal theory . . . 61

KARLIN, S., and J. McGREGOR: Direct product branching processes and related induced Markoff chains. I. Calculations of rates of approach to homozygosity 111

KITAGAWA, T.: Automatically controlled sequence of statistical procedures 146

LE CAM, L.: On the distribution of sums of independent random variables 179

Los, J. : Limit solutions of sequences of statistical games . 203

PITMAN, E. J. G.: Some remarks on statistical inference 209

STEIN, C. M.: Approximation of improper prior measures by prior probability measures •......•.••.........•. 217

Y AGLOM, A. M.: Stationary Gaussian processes satisfying the strong mixing condition and best predictable functionals . . . . . . . . . . . . . . . 241

Y AGLOM, A.M.: Strong limit theorems for stochastic processes and orthogonality conditions for probability measures . . . . . . . . . . . . . . . . . 253

Unless otherwise indicated, all the papers in the present volume

were prepared with the partial support of the

U.S. National Science Foundation, Grant GP-10