of 8
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
1/8
.
SO M E G UIDELINES
FOR
THE
EVALUATION
OF
NUCLEAR DATA
Donald L. Smith
Techno logy Development Division
Argonne National Laboratory
Argonne, Illinois 60439
March 20
1996
Introduction
Mo dern data evaluation methodology draws upon basic principles from statistics.
It
differs from earl
ad
ho
approach es which are completely subjective (e.g., eye guides to data) or are o bjectiv e in a limited sen
(e.g., combinations
of
reported data by a simple least-squares procedure w ithout regard to correlations in t
data errors
or
a ca reh l scrutiny of the data included in the evaluation). In a ddition to utilizing more rigoro
mathematical procedures, m odem evaluation methodology involves taking great care to insu re that the d a
which are being evaluated are equivalent to what has been assumed in the evaluation model and th at the v alu
are con sistent with respect to the use of standards and other hn da m en td physical parameters. This sho
memorandum cannot substitute for more comprehensive treatments of the subject such as can be found in t
listed references. The intent here is to provide an overview of the topic and to impress upon the reader th
the evaluation of data o f any sort
is
not a straightforward enterprise. Certainly evaluations cannot be carri
out automatically with computer codes w ithout considerable intervention on the part of the evaluator.
There are two types
of
information (da ta). One is ob jective data based
on
experimental m easuremen
The other is subjective data which, in the case o f basic nuclear quantities, often em erge from nu clear mod
calculations. It is rare that there is sufficient experimental information upon wh ich to base a c omprehensi
evaluation. Usually it
is
necessary t o merge the complementary processes ofmeasurement and modeling
order to generate such an evaluation. Furthermore, various nuclear quantities are not independent. Fo
example,
an
evaluated file for a particular isotope o r element, as it appears in
ENDFB
or any other nation
or international file, consists o f many interrelated componen ts (e.g ., partial cross sections) correspon ding t
various reaction channels. Partial cross sections must add up to the total cross section. Unitarity of the S
matrix appearing in theoretical calculations generally insures that this w ill be the case when these q uantiti
are derived from nuclear models. However, this will not happen for experimentally derived quantitie
Completely different experimen ts and techniques are involved in measu ring individual partial cros s section
(or
comb inations thereof), often leading to a rather messy state of affairs for the evaluator to
sort
out
carrying out an eva luation. What is measured is rarely equivalent to what one seeks t o obtain. T he relationsh
between what is measured (or calculated) and what is sough t must be spec ified in order to carry out a prop
evaluation. The experimenter or model calculator ought to be aware of this , but frequently this is not t he ca
so it is left to the evaluator to bridge the gap in understand ing. Ideally an evaluator ought to be w ell verse
in
all aspects
of
nuclear model calculations, nuclear data measurements and the analysis
of
measured data s
that all
the
features of the raw materials which must be employed in his evaluation are well understoo
Realistically this happens rarely,
so
comprehensive evaluations such
as
those appearing in
ENDF
are often th
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
2/8
DISCLAIMER
Portions of this document may be illegible
in electronic image products. Images are
produced from the
best
available original
document.
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
3/8
DISCLAIMER
This report was prepared
as
an account of work sponsored by an agency of the
United Stat es Government. Neither the United States Government nor any agency
thereof, nor any of their employees, makes any warranty, express or implied, 3r
assumes any legal liability or responsibility for the accuracy, compieteness,
or
use-
fulness of any information, apparatus, product, or process disciosed, or represents
that its use would not infringe privately owned rights. Reference herein to any
spe
cific commercial product, process, or service by trade name, trademark, manufac-
turer, or otherwise does not necessarily constitute or imply its endorsement, recom-
mendation, or favoring by the United States Government or any agency thereof.
The views and opinions of authors expressed herein do not necessarily state or
reflect those of the United States Government or any agency thereof.
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
4/8
result of collabo rations involving individuals with various complementary skills. This used to be feasible i
house at many of the individual laboratories in the U.S . doing nuclear data research, since in earlier times t
resources available were far more extensive than they are now. Due to staffreductions, retirem ents, la borato
closings, etc. ,
it
is far less commo n now to find under
one
roof all the necessary skills needed to perform
comprehensive evaluation properly. Today inter-laboratory collaboration is essential. Adequate hndin g is al
required to supp ort the personnel involved in this labor-intensive activity.
Modern Theory
of
Data Evaluation
In rather abstract terms, the process of data evaluation reduces to the follow ing: Give n a d ata set B (whi
may include both objective and subjective information), determ ine what is the m ost likely (best) set o f valu
for the evaluated quan tities represented by a vector p
=
(p1,p2,
..
pk, ... pK).The m ethodology describ
here is based on the application of three hndamental principles: i) Bayes' Theorem, ii) the Principle
Maximum Entropy, iii) the Generalized Least-squares Method. These principles
are
somewhat interrelate
as discussed in the references below . The following formalism is a h ll y probabilistic one in the sense that
offers a prescription for generating a probability distribution hn ct io n p (p ) that embodies all the informatio
available concerning the parameters p .
Bayes' Theorem and the Principle of Maximum Entropy
In the p resent context, Bayes' Theorem assumes the form
where pa( p) is the priori probability distribution that describes the knowledge of p before any ne
information is acquired, B represents the newly obtained information, I(dlp) is the likelihood that th
parameters p could have led to the data set B p( pp) is the ap os ter ior i probability distribution for p (after th
new information became available) and C is a positive constant which insures that the ap ost er io ri d istributio
is no rmalized, i .e., that the requirement J'p(p1B)dp = 1 is satisfied when integration
is
carried ou t over th
entire space of physically reasonable parameters p .
Suppo se that the experiments and/or calculations which generated the data set
B
involve a collection o
J physical quantities denoted collectively as
y
= (y1,y2y
..
,yj,
...
,y,). The generation
of
data entai
uncertainties, therefore let
Vy
represent the covariance matrix (error matrix) for these data. Thus, B
represented by the values
(y,V,,>.
t is assumed that given param eter set p it is possible to calculate a set o
J quantities f(p)
=
[f,(p),f,(p), ... ,q(p), ... ,f,(p)] which are equivalent to the data values y (one-to-one). Th
Principle of Maximum E ntropy enables a relatively simple expression for
L ~91p)
o be written down directly
namely,
where
+
signifies matrix transposition and -' ignifies matrix inversion. Vy s required to be positive definit
If the priori know ledge includes a parameter set pa and corresponding positive definite covariance matri
V,,
then the Principles
of
Maximum Entropy state s that
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
5/8
which is a multivariate normal distribution.
The Generalized Least-sauares Met hod
The Generalized Least-squares Method (GL SM) follow s from imposing a maximum -likelihood conditi
on Eq. ), namely, that the GL SM solution for p is the one for which the posteriori probability distributi
achieves its maximum value. Because o f the nature of the exponential function, combining Eqs.
(1)-(3)
lea
to the requirement
provided that the new and prior knowledge are essentially independent (a point which an evaluator mu
always keep in mind when collecting input data and prior information for a
GLSM
evaluation). If t
relationship between
p
and f p) is non-linear,
in
general it can be quite difficult to find a solution whi
satisfies Eq. 4).There are ways t o do this based o n numerical integration involving the probability distributio
but this will not be discussed here. However, if the model is linear, i.e., if f(p)
=
Ap, then the posterio
probability d istributionp(pIB) is a multivariate normal distribution. T he m atrix
A
is often referred to as t
design (o r sensitivity) matrix. A comp lete description of the relationship between the acquired d ata and th
parameters to be derived from the evaluation process is contained in
A.
Even if the relationship between
and
f p)
is non-linear it may still be possible to linearize the problem via the ap proxim ate relationship
where the elements of matrix A are given by the expression ajk = [d{/dpk] evalua ted at p = pa. Th
approximation
in
Eq.
(5)
is vaIid as long as the solution p does not differ too much from the prior estima
pa. In practice, most evaluations rely on being able to use this approximation, and therefore experience
evaluators try to set up an evaluation process
so
that this condition is reasonably well satisfied.
For the linear (or "linearized") model, the solution t o
Eq. 4) s
contained
in
the following four equatio
which form the basis of data evaluation by G LSM :
Q
=
AVaA ,
7
Two
features of this solution are worth pointing out here. First, the solution yields not only a param eter vecto
p (the evaluation itself) but also a corresponding covariance matrix V, representing the uncertainties in th
evaluated quantities. Second , there is a statistical test provided gr tis
in
the
form
of the quantity
(x2)-.
Th
quantity obeys a chi-squared distribution with J degrees of freedom . comparison with standard tables of th
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
6/8
chi-square distribution enables the evaluator to determine whether the input data and/or evaluation model
consistent. If inconsistencies are found then the input information and evaluation model m ust be exam ined
t r y
to d iscover the source of the problem.
Practical C onsiderations in Data Evaluation
The procedure sketched out above is deceptively simple. In order to emphasize this point it is worthwh
examining each o f
the
quantities appearing in Eqs.
6)- 9).
p (parameters t o be evaluated):
It m ay seem obvious what it
is
that one wishes to evaluate but this is not always the case. For examp
if
an evaluated reaction cross section
is
desired vs. incident energy, this is really a co ntinu ous fimction. Ho
should it be represented? One approach is to give "point" cross sections, namely, a set of energies a
corresponding cross section values such that one can reconstruct the desired curv e through interpolatio
Anothe r app roach is to give group cross sections, namely, interval-average cross sections for well-defin
energ y intervals. If the desired quantity is a derived value, e.g., a Maxwellian-spectrum-average aptu
neutron capture cross section then this needs to be well defined before the evaluation process begins.
pa
and
V,
(prior param eter values and their uncertainties):
In the GLSM method it is necessary to start from som ewhere, even if it is only a guess. The pri
parameters
pa
might be values from an earlier evaluation (in which case the new evaluation should include on
information not reflected in the earlier evaluation) or they m ay result from model calculations which are
be "ad justed" by the inclusion of new experimental data via the GLSM method (data merging ). Th e as socia t
covariance matrix V, needs to be gene rated in a consistent way (e.g ., it must be positive definite). This is n
easy to do
if
the prior values are merely estimates, or
if
they are based on calcu lations using m odels that a
very sensitive to fbndam ental nuclear interaction constants and that are not well validated to b egin with.
seems rather intimidating to be forced to provide something as input to the cod es which implement GLS
in the face of such sketchy knowledge. However,
it
should be comforting to know that assumed prio
parameters with large errors gene rally carry very little weight in the GLS M process, and the solution ten
to be heavily dom inated by the new information if that is both ex tensive and relatively accu rate. Still, th
happy state of aEiirs can be thwarted if the corre lations existing in the covariance matrices
V,
and Vyare to
strong, posing yet another po tential pitfall for the wary evaluator
y
and
Vy
(new data and their uncertainties):
The most important thing to know here is what the data actually represent. Are the energies we
established? What was the neutron spectrum
in
which they were measured? What standards w ere used? A
the various data collected from the literature truly independent or a re there com mon sourc es
of
uncertaint
These and many other questions force the evaluator to examine the data and their documentation ve
carefully, and it is often necessary to adjust these data for changes in standards, to transform t o new energ
grid points, e tc. This process of adjusting data prior to their evaluation is the most time consuming part
evaluation work, and often it
is
the most arbitrary one since poor documentation of published data is
notorious problem . Only when the input data are properly prepared can one hope to get reasonable resul
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
7/8
from
an evaluation, regardless of the procedure used. This simply cannot be do ne
by
a "machine approac
without the aid of human scrutiny.
f p) or A (the m odel which relates the data to the evaluated parameters):
This is a test of the evaluator's skill. The elements of matrix
A
can be generated easily enough from t
selected model, either analytically o r via numerical procedures. What is taxing is knowing ju st how a giv
piece of data relates to the parameters to be evaluated when the data in question a re either undocumen ted
relatively poorly docum ented. Often it
is
necessary to reject certain data p oints because crucial informati
is lacking. For example, if a cross-section published in 1957 indicates an energy " 14 MeV " could this me
13.9MeV or 14.1 MeV? If the physical quantity is known to vary rapidly with energy this is a crucial m att
Often the evaluator has to look at the o riginal paper and, fiom a description of the experim ental setup, try
answer the question. Early works fi-equently ail to indicate which standards were used o r t o give actual valu
for these standards when they are men tioned. Based o n the date of the wo rk and clues in the documentati
it may be p ossible for an evaluator to estim ate what w as used in the original data analysis with reasonab
reliability. Frequently that is no t possible. If care is not taken to relate what w as m easured t o w hat
is
soug
then the evaluation process reduces to an exercise not unlike that of comparing apples and oranges.
x2)> , , parameter (test for confidence in the G LSM evaluation):
If
all the data are reasonably consistent with the assumed uncertainties, and if the evaluation model
consistent with the input data, then
(x >,, =
(number of degrees of freedom) should result
from
the analys
embodied in Eqs. (6)-(9). If
(x2)
>> J, then there are inconsistencies which need to be resolved by th
evaluator. This may entail lookingat
all
the data sets to see if they are discrepant
or if
the assumed errors
a
too small. It may also entail looking at the evaluation model which relates the da ta and param eters to see
it
is
som eho w faulty. In any case, the evaluator must do something n evaluation with a low degree
confidence (large chi-square value)
is
simply unacceptable.
Finally, it should be mentioned that computational round-off errors assoc iated with th e adjus tment of da
or with the GLSM evaluation process (which often involves the inversion of large matrices) can lead
inferior evaluated results. Evaluators need to insure that their analyses are carried out using adequa
numerical precision.
Summary
Data evaluation, like making goo d wine o r cheese, involves not only goo d quality ingredients but als
depends critically on the
art
of the evaluator". Combing the literature and experime ntal data files for the ra
materials needed in evaluations has been likened to archaeology .
A
good evaluator must be a very patie
individual. Mo dern data evaluation concepts, as embodied in GLSM, provide an unbiased approach to th
merging of all types
of
data which become
known
to an evaluator, once it has been assembled, examine
critically and put into a unified format for analysis. There are various codes that can do the actual GLSM
calculations, depending upon the nature of the data (e.g., SAM MY, G LUCS , GMA , GL SMO D, UNFO LD
BAYES,
etc.). T he particular software which
is
used is generally of less importance than un derstanding th
nature o f the d ata em ployed and verifying its fidelity.
8/10/2019 Donald L Smith Nuclear Data guidelines.pdf
8/8
I
References
This list gives a g oo d starting point for a more extensive study of the literature on this subject.
1
2.
3.
5
6
Donald L. Smith, "Covariance Matrices and Applications to the Field of Nuclear Da ta", Repo
ANL/NDM-62 Argon ne National Laboratory
(1
98
1).
[A basic primer on the concepts of modern data evaluation. Formulas are derived and some simp
examples are w orke d through in detail].
Donald L. Smith, "Non-evaluation Applications for Covariance Matrices", Report
ANL/NDM-6
Argonne National Laboratory (198 1).
[Extends the material presented in Report
ANJNDM-62
by giving numerous simple examples
fro
everyday nuclear applications including detector calibrations, etc . Note that the methods used in analyzi
experimental nuclear data are identical to those employed
in
nuclear data evaluation].
Donald L. Smith, Probability, Statistics, and Data Uncertainties in Nuclear Science and Technolog
American Nuclear Society, LaGrange Park, Illinois (1 991).
[A comprehensive treatment of basic probability theory and the development of modern nuclear da
analysis and evaluation methodology. Numerous simple examples are given and a detail search of t
literature on nuclear data evaluation methodology up to 1990 is documented
in
the reference list. Availab
in hardback (269 pages) &om the American Nuclear Society Press, L aGrange Park, Illinois 60525, USA
Price is 25 with a 10% discount available for N S members (credit card o rders accepted).]
Nuclear D ata Evaluation Methodology, ed. Charles
L.
Dunford, World Scientific Press , Singapore (1993
[ n
extensive collection of papers on nuclear data evaluation reflecting the s tatu s of development up
1992. Contributions t o a conference on data evaluation held at Brook haven National L aboratory.]
DonaId L. Smith, "A Least-squares Computational 'Tool Kit"', Report ANL/NDM-128, Argonne Nation
Laboratory (1993).
[A handy reference on the basic principles of data evaluation by the least-squares method (both simple an
generalized). Exam ples are given and computer c odes that are usefbl
for
data analysis and evaluation a
described.]
A.
Pavlik, M.M .H .Miah, B. Strohmaier and H. Vonach, "Update
of
the Evaluation of the Cross Sectio
of
the Neu tron Dosimetry Reaction 03Rh(n,n1)103mRh 1,eport INDC (AUS)-015, IAEA Nuclear Da
Section, International Atomic Energy Agency, Vienna (1995).
[Well-documented description of a recent evaluation effort at the U niversity of V ienna. The procedure
associated with collecting data fiom the literature and reviewing and adjusting it in preparation
fo
evaluation by the Ieast-squares method are discussed very well
in
this report.]
