Revista Mexicana de Física 35 Suplemento {1989} 580-588

Are hidden variables possible 7*

Abstract. The impossibility proof givcn by von Ncumann for hiddcnvariables to complete quantum mechanics is a direct consequence of thedensity-matrix formaJism; since this stand s or falls •.••.ith quantum theor)'itself, "deterministic" or dispersionless hiddcn variablcs, but onl)' thesc,fiust be ruled out. This is sho•.••.n to bc valid for all statistical theoriesin physics. That there exist theories •.••.ith acceptablc hiddcn-variablestructures then shows that dispersive hidden variables rannot be rllledout either classically or qllantum-mcchanically.

As 500n as lhe essentially slatislical nalurc of quantum mcchanics was recognizeJ,thc apparcntly inevitable dispersions predictcd by thc throry and observed in cxper-iment were thought by many authors to be cxplainablc in terms of furthcr dynamicalvariables, commonly called "hiddcn" in view of liJeir unknowll nature. By atlribul-ing different values to thcse variables for differcnt systems in an experimental serieswhere the \Vavc function was the same for all systems, it was hoped lhat olle couldaccount for the fact thal the choice of one among scveral possihle cigenvalues uponmeasurcmcnt appcared to be random. The discussion was stilk..¿ for a time by vonNcumann 's proof PI that sueh hiddcn variables could nol consistcnlly be introducedsince "lhc prcsenl system of quantum mechanics would have to be ohjectively falsc,in order that anothcr dcscription oC the elementar)' processcs other lhan the statis-tical one be possihlc". \Ve propose to show here. in part from von Neumann's OWII

argumenl, that this conclusion in lIndllly pessimistic.The problcm was rMpened by Bohm1s successful constrllclion [2] oC a modcl

explicitly containing hidden variables that ne\'erthcless exactly rcduccd to qllan-tum mechanics. The modcl pro\'ed nol to be a good physical t!leory in the senseeilher oCinherent plausibility or oC experimental \'erificalion; all the sarne il pro\'('dimporlant simply hecallsc for lhc first lime an unobjcelionable cOllllterexamph'to von Neurnann 's argllIllent could be exhihited. This is aH to lhc good: <}uantulllmechanics is at lhc sarnc time so sllcccssful and 50 mllch plaglled by unresolved con-ceptual problems lhat a fundamental qucstion like thal of hiddrn variahles shollldBot simply be shclved. Yet the von Neumann proof led many aulhf)rs (illcluding\'on Neumann himsclf) to interprct the statistical nature of <}uanturn ITlechanics asirreducible and in conniet with the causal conceplion underlying c1assical physics.The unacccptability of lhis conclusion has led others to doubl the validity oC vonNcumann's proof; thus de Broglie [1]1Bell [4] ami, more r('('('nllYl Bilsakis ¡51."Written during a lea .•.e of abst"nce al J)('tnoCrilos Centre of Scif'ntific H('scarch. Attiki8, Greece,

in 1985.

Are hidden van'ables possib/e? 581

Now \'on Neumann lS proof did contain certain weakncssesj but we shall argueherc that rcjecting it thereforc is nol possible without also rejecting quaotum me-chanics ilself. lIowever, al least somc of the weakncsses in the proof havc sincebeco O\'crcome (sce Del! [4] for a review and aH rcfereoccs). \Ve show here

1fur-

thermorc1 that tIJe quantum case gencralizcs lo aH c1assical slatistical throrics. Dulsuch thcorics nol only exist bllt posscss weJl underslood "'hidden" variables. Thisapparent conlradiction is resolved by corrcetly interpreting what is proved

1in the

SCIISC lhal any hidden variables Illust thcmselves be statistical in nalure and possessnon-negligible dispersions; lhe)' may neverthclcss be given fixed values

1but then

an enlircly differenl lhcory results. This inlerprelalion applies lo von Neumann lSargumenl; so far from constituting an impossibilily proof, it should thcrefore besecn as giving a slrong hint aoout lile structurc of hiddcn-variable thcorics lo bedeveloped for quanlum IIlcchanics.

JI

Von Neulllann proves that one cannol add hiddcn variables so as to render quantumIllcchanics "'deterlllinislic" as a ("orollary lo olle of the main theorems in this book (1).This theorcm establishes lhe cxistcnce and principal propcrly of density matrices.\Ve q1l0le:

TIIEOREM DM: To cver)' quantum-nwchanical ensemble thcre corrcsponds a linearscmi-definitc I1crrnilan matrix p = [PT1InL sllch that the cxpcetationvalue of an)' operator Ji corrcsponding to a physical qualltity is:

mn

(\Ve note that "'observable" is generally usoo now for von Neumann1s "'physical quan-tity".) From TIIEOREM D~I thc rcquircd corollary is dcrived in a mathematicallyullexceptional way. It states:

COROLLARY BV: No qllantum-mechanical ensemble descrihed by a dcnsity malrixsatisfying the requirements of TIIEOREM D~f is dispersion-frcc.

Here an ensemble is tcrmcd dispersion-frce if for aH opcrators il wc have that (R2) ;;;;;:(il}2. On the other hand1 pure or homogeneous ensembles of course exist, whichcannot be split into subensembles of different expectation values; these correspondto vectors in thc underlying Hilbert space. The details of the proof, with usefulcomments, are given by Albertson [6); the thoory oC density matrices will be fOUDde. g. in Fano [7).

Now the proof von Neumann gives of TIIEOREM DM, though m&thematiCA1.lycorrect, rests 00 APostulAateswhich may with good rcasoo be doubted. Tbus thepostulate that if A and B are the quantum operators that-eorrespond to the cl.uaicalobservables a and b, the operator corresponding to a + b will be Á + !J, cve:n if A

582 T.A. Brody

and B do not commute ..A. Bell di.cusses [41, thi. cannot alway. hold. For thi. andrelated rea50DS, the von Neumann proor is nol realIy acceptable.

Bul one canDot conclude th3t the theorern, and henee ¡ls COROLLARY HV, isnot valid, and this for two rcasaos:

(i) Gleason (8) has given an alternative proof, at least for Hilhert 'paces ofdimensionality greater thao 2, that escapes sorne oí the criticisffiS made oC vonNeumann's proof, while Kochen and Specker [9) provide yet a third proaf which,while also Dot unobjectionable, resists yet other criticisms;

(ii) It .hould have been obviou. tbat the place occupied by TllEOREM DM in vonNeumman's foundational structure rcodees it indispensable. Van Neumann [1] offerstwo a1ternative postulational schemes COI' ..:¡uantum mechanics: one, the Hilbert-spaceformalisffi, in which each vector in a finite or infinite-dimensional Hilbert spare cor-tesponds to a quantum statc, i. f. a normalisable solution oC the appropriate waveequation: and two, the density-matrix formalism based.on THEOREM DM. He shows,moreover, that these two formalisms are intimatcly relatcd. By adding the plausiblestatistical postulate that for a mixture of states no interference terms between ele-ments of the mixture should arise in any expectation values, one obtains from theHilbert-space Cormalisma Cormalismexactly equivalent to the density-matrix one;and the subdass oC homogeneous density matrices, i.f. those Cor which p2 = p, turnsout to be equivalent to the set oC normalized. vectors in the corresponding Hilbertspace. If V represents the dass oC aHdensity matrices, I the subclass oC idempotentones, r the ray space oCnormalized wave Cunctions,and S'P the statistical postulate,ane may write, symbolicalIy,

r '* I e v,V= r +SP.

(1 )

From THEOREM DM the impossibility of non-dispersive ensembles follows atonce. If this also makes hidden variables imp05sible, as von Neumann contends,then indeed TllEOREM DM must be wrong before hidden variables can be accepted.But if we abandon TllEOREM DM as the basi. for the den.ity.matrix formali.mfor quantum mechanic., then the two relation. (1) imply that the Hilbert-.paceformalism must also be abandoned. The introduction of any hidden variables isthen nugatory.

The. dilemma is inescapable so long as one forgets the Cactthat the impossibilityof non-dispersive ensembles (which we may now accept as a well-established conse-quence oC the quantum formalism) only rules out non-dispersive hi¿den variables.Indeed, von Neumann himself states clearly and repeatedly what is orten over-looked, namely that the hídden variables he is concerned. to prove inadmissible arethe non-dispersive ones whose incorporation would eliminate the statistica1 elernentfrom quantum mechanics and reduce it to a dassical theory of the type wherethe theoretical predictions correspond (to within experimental error limit.) to theindividual measured values. But thi. aim i. meaningful only because, as Bit.aki. (5]points out, von NeumaDD confuses causality with determinism and hence does not

Are hidden variables possiblef 583

even consider dispersive hidden variables¡ for if hidden variables are themselves ofstatistical chara.cter, they will not eliminate the statistical elements from quantumtbcory nor render it deterministic a.sfor instance few.particle mechanics ¡s; but theywill make patent that quantum mechanics and causal description are compatible.But it is this that hidden.variable theorists wish to achieve, and not (as is oftenalleged) a returo to c1assical theory. The statistical nature of quantum mechanicscanDot be explained away without making it incompatible with experimental fa.cts;bul il can be'exlllained.

We proceed lo spelllhis oul in furlher delai!.

III

That von Neumann's result is more general may be seen by deriving it from pral>.abilily lheory.

A physical theory must possess dynamical variables which satisfy equations ofmotion (when the independent variable is time) or other similar equations. Rete weneed only consider a single variable, say x. lf the theory is satistical in nature, xwill posses a cumulative probability.distribution funetion p(x), and the equations oímotion will describe the time evolution of p(x). If now a further, "hidden", variabley is lo be added lo lhe lheory wilhoul cbanging ils eariier resulls, lhen firstly ajoint distribution function Q(x,y) must exist, and secondly its marginal must beP(:r):

P(:r) = Q(:r,oo).

The distribution function of y is given by

R(y) =.Q(oo,y).

(2)

(3)

lf y is lo be "delerminislic", il will lake fixed value ~, while R(y) musl be lheHeaviside step function:

R(y) = H(y - ~), dR(y) = 6(y - ~)dy. (4)

From lhe definilion of a condilional probabilily we have

Q(:r, y) =¡~P,(:rIY') dR(y'), (5)

where P,(:rly) is lhe probabilily lhallhe dynamical variable lake lhe value:r or less

584 T.A. Brody

wheo the added variable has the value y. Theo

Q(x,y) = {OP,(xl~)

y<~

y>~(6)

and the value at y = ~ may be takeo either as Oor as P,(xl~). Usiog Eq. (2) yields

aod combioiog (6) with (7) ooe has

(7)

Q(x,y) = {OP(x)

With (4) this gives

y<~

y > ~(8)

Q(x,y) = P(x)ll(y). (9)

Eq. (9) eslablishes that a delerministic addcd variable, in lhe sensc slated aboye,is statistically independent oC the original variable x. This is compatible w¡lh itsbeing linked lo x through aD equation oC motion or ils equivalent, and it cannoltherefore be'iotegrated ioto the strueture of the theory for x. Thus Eq. (9) is thegeneralization lo any slatistical theory oC von Neumann's proaf that deterministichidden variables cannol be introduce<! ¡nio the structu~c oC quantum mechanics.

Three comrnenls should avoid sorne misundcrslandings.i) Ifthe distrihution oC x is a)50 a Heaviside fUDclion, Eq. (9) is valid bul ¡rrelevant.For in this case lhe theory that contains x is no longcr a statistica! theory.

ii) The arguments leading to Eq. (9) do not affcct any paramcters in the thcory orin any rnodel built on it; whether already in the original theory or introducedalong with y, their distribution is always the IIeavisidc step function.

iii) The argument has been stated in lerms oC a single fixcd value Cor y but caneasily be extended to the case oC y non-zerD in an intcrval. Instead oC Eq. (4)one has

ll(y) = {O Y < ~1 Y > ~'

(4')

while R(y) increases monotonically in the intcrval (TI, 1,'). Eq. (8) now beco mes

{

O y<~Q(x,y) =

p(x) y > ~'(8')

A~ hid1en variables possible~ S85

In the interval (~, ~') sorne different expression holds. Eq. (9) is to be replacedby

Q(x,y) = P(x)R(y) (9')

If the interval (1],1]') is small, say oC the order of the experimental error limits ony, the incoQlpatibility oC y with the existing thooretical structure is estabIishedto within thaL error limit. Out iCall physically relevant values oC y CaHinside(1],1]'), then y"is not dispersionles~. (9') does not hold in the region oí interest,and the present argument does not impede the introduction oí y into the thoory.Intermediate cases must be examined on their merits.

IV

The conclusions drawn rrom (9) resp. (9') c1early hold ror a classical theory suehas statistica1 mechanics. This is in apparent contradiction with the Cact that herethe hidden variables exist and are described by classical Newtonian mechanics andwouId generally be caBed deterministic. The difficulty is due to a conceptual confu.sion Costered by the unclear way in which the Coundations oC statistica1 mechanicscommonly are presented in physics textbooks.

A description oC an N-particle system sueh as a volume oC gas is perfectlypossible, at Ieast in principIe, within the Cramework oC classical mechanics. Theproblem oC finding the initial positions and velocitics (which is oCten, but withHttle justification, cited as the reason Cor the statistical approach) can in manycases be solved, Corinstance in the case oC medium-sized stellar clusters or in thatdf a computational model. It would dcpend entirely on the initial conditions andtherefore differ greatly íor two identical volumes oC gas at the same temperature andpressure, or for thc SaIne volume in thermal equilibrium at different times. What issought, even in computer models, is a description oC such many.body ~ystems thatis independent of the initial conditions; this is not possible in c1assical mechanicsbut is achicved by averaging over aH possible initial conditions (or a statisticallyadequate subset oC them). It is not the size oí N but this need to eliminate the initia1conditions that justifies the approach oC statistical mechanicsj indeed, statisticalmechanics provides a powcrful tool for dealing with systems oC small N, weHwithinour present powers, if not of analytical solution then oC a computationaI one. But inthe ensemble of possible trajectories over phase space the hidden variables do havethe dispersions nceded according to the conclusions oC the preceding section.

The large-N justification is not always wrong; but it springs Crom a differentconsideration, that oC the relativc size oC the Uuctuations about the ensemble av.erages. When thcse fiuctuations are small compared to the relevant observationa1errors, thc ensemble thcory offcrs much greater physical insight as well as an easierapproach to applications. \Vhen they are much larger, the c1assical theory is oftenpreíerable. Now in many cases the rclative size of the fiuctuations is oC the order oCN-1/2j hence a secming justification oC the statistical approach when N is large.

586 T.A. Brody

The case oí statisticai mechanics also offers a good illuslration oC the furtherpoinl that while the hidden variables roe a statistical theory must themselves beslatistical, they can also be "deterministic" in the sense oí having zero dispersions;bul the theory based 00 them has then a wholly d¡fferent character, as incompatiblein conceptioo with the statistical theory as in raet classical and slatistical mechanicsare. The ensemble concepl and the proc<lSS oC averaging ayer the ensemble cslablishthe passage from the classical lo the statistical theory; the ¡nverse passage is nolpossible.

Ignoring the distinction belween the two theory types leads lo 3uch errors asstipulating that the dassieal hidden.variable tbeory should reproduce the algebraieslrudure oí the statistical theory. Von Neumann's additivity postulate is of thiakind, as Bel! [4J shows by means of an il!uminating example. A similar problemarise5 in the impossibility proof of Koeben and Speeker [9): they consider the fune-tion fA that, for a given operator Á, leads from the hidden-variable apare n to thepredictions

(A) = lo J.•.(w) dp~(w).

They then require that

The authora\ juatification [9] is that "In any theory, one way of measuring A2 con.sists in measuring A and squaring the resulting value. In fact, this may be usedas the definition oí a fundian oí an observable." This coníuses the two levels.The theory predicts (A), whose square is (A)'. Experimental!y, one finds indi-vidual observationa, which are not predicted by the theory (only their averageis), oue squares them and then averages. If there is dispersion the two resultadiffer.

vThe dassiea! impossibility proof for bidden variables, eonsequent upon Eqs. (9)and (9'), may then be interpre ed in the fol!owing way:

[f hidden varia.bles are to be introduced in a lltatistical theory in orderlo obtain a. more deta.iled description, they must themselves be disper-sive. The sa.me variables may also figure in non-statistica1 form in anothertheory, which however will be of ditrerent type. (10)

Formulating the conclusion in this way underlines ils positive value inatead oCexhibiting only wbat it forbids.

Are huMen van"able8possiblef 587

Conclusion (10) is straightforwardly applicable to quantum mechanics. As wasseen in section 2, the von Neumann prooC leads to the same conclusion for quanturnmechanicsj it may therefore be scen as a speciaI case of (10).

Since quantum mechanics is a dynamical thcory, (lO) strongly suggest that anyhidden-variable formulation must be a stochastic one in order to be successful. Thatthis hint is in the right direction is oonfirmed by the faet that not only Bohm's [21original hidden-variable theory but alllater ones (Bohm and Bub [IOJor Wiener andSiegel [11) are good examples) are of statistieai type and involve stochastic variables.The same is true of Nelson's [12] much more complete formulation. Whatever thevalue as physical theories oC these extensions oC the quantum formalism, they exhibitthe capability of hidden-variable theories to provide new interpretative elernents ofmuch significan ce.

Qne cannot, however, conclude that adding hidden variables to the quantumforrnalism is the way out of the conceptual confusion that surrounds quantummechanics. As we have scen dispersive variables could indeed be added¡ but tbeyare undesirable, bccause rather than patching up an existing theory ODe shouldresolve the difficulties by crecting a conceptually more coherent thecry on a newand physical1y more satisfactory basis. It may have becn sorne sur.h considerationthat kept Einstein, otherwise a vigorous critic of the deficiencif...i of the standardview, from giving his approval to hidden-variable theories. And it is certainly newtheorization that has in recent ycars Icd to the devclopment of the idea that theamplitudes of the random background in the electromJJ.gnetic field could play therole of hidden variables. The quantum formalism is only a very good approximationto the exact ensemble avcrages oCthe motions vf charged particles in such randornfields, and so a theory 00 this basis is oot simply a hidden-variable extension ofquantum rnechanics. The hidden variables may also take non-f1uctuating values: wethen recover classical electromagnetic theory, in which quantum behaviour appears.For a recent review and referenees e"" Brody [13J.

VI

1 would like to thank L. de la Peña for several stimu1ating discussions, and A.K.Theophilou, !leadof the Department of Theoretical Physies at the "Democritos",both for extended hospitality and for many helpful comments.

References

1. J. von Neumann, Mathematisehe Grundlagen der Quantenmechanik, Julius SpringerVerlag, Derlin 1932 (Engl. tr. Princeton University Pres5, Princeton, N.J. 1955).

2. D. Bohm, Phys. Rev. 85 (1952) 166, 1BO.

3. L. de Broglie, La physique quantique ~stern.t.elle indéterministe~, Gauthier.Vill&n,Pan, 1953.

4. J.S. Bell, Rev. Moo. Phys. 38 (1966) 441.

588 T.A. Brody

5. EJ. Bitsa.kis, to be published.

6. J. Alberlson, Amer. J. Phys. 29 (1961) 478.

7. U. Fano, Rev. Mod. Phys. 29 (1957) 74.

8. A.M. Gleason, J. Malh. Mech. 6 (1957) 885.

9. S. Kochen and E.l'. Specker, J. Malh. Mech. 17 (1967) 59.

10. D. Bohm and J. Dab, Rev. Mod. Phys. 38 (1966) 453.

11. N. Wiener and A. Siegel, Nuovo Cim. Suppl. 2 (1955) 3Q.tjA. Siegel and N. Wiener,P~ys. Rev. 101 (1956) 429; N. \Viene< and A. Siegel, l'hys. Rev. 91 (1953) 1551.

12. E. Nelson, Phys. Rev. 150 (1966) 1079; Dynamical Theorie~q oJ Brownian Motion,Princeton University Press, Princeton, N.J. 1967.

13. T.A. Brody, Rev. MeE. Fís. 29 (1983) 461.