ww

w.in

l.g

ov

Benchmarking Experiments for Criticality Safety and Reactor

Physics Applications – II –Tutorial

John D. Bess and J. Blair Briggs – INL

Ian Hill (IDAT) – OECD/NEA

This paper was prepared at Idaho National Laboratory for the U.S. Department of

Energy under Contract Number (DE-AC07-05ID14517)

2012 ANS Annual Meeting

Chicago, Illinois

June 24-28, 2012

Outline

I. Introduction to Benchmarkinga. Overviewb. ICSBEP/IRPhEP

II. Benchmark Experiment Availabilitya. DICE Demonstrationb. IDAT Demonstration

III. Dissection of a Benchmark Reporta. Experimental Datab. Experiment Evaluationc. Benchmark Modeld. Sample Calculationse. Benchmark Measurements

IV. Benchmark Participation

2

DISSECTION OF A BENCHMARK REPORT

3

© Microcosmos

Benchmark Evaluation

4

Format guides provided on handbook DVDs

Why Such a Rigorous Format?

• These are Handbooks or Reference Books– For the benefit of the user

– Orderly layout to assist the user

– Information is always in the same location

– Information has been rigorously verified

• Separation of Geometry, Materials, and Temperature

– Neutronics computer code input

– Allows for systematic and detailed review/verification

• Not a Compilation of Technical Reports

5

6

• Experiment title

• Identification number(Fissile Material) - (Physical Form) - (Spectrum) - (Three-Digit Numerical Identifier)

Subcritical measurements are denoted by including the letters “SUB” at the beginning of the identifier.

• Key words

A list of words that describe key features of the experiment is provided.

ICSBEP Content and Format

Fissile Material Physical Form Spectrum

Plutonium PU Metal MET Fast FAST

Highly Enriched Uranium HEU Compound COMP Intermediate-Energy INTER

Intermediate Enriched Uranium IEU Solution SOL Thermal THERM

Low Enriched Uranium LEU Miscellaneous MISC Mixed MIXED

Uranium-233 U233

Mixed Plutonium – Uranium MIX

Special Isotope SPEC

7

ICSBEP Content and Format (Continued)

1.0 DETAILED DESCRIPTION

• Detailed description of the experiments and all relevant experimental data

– Who, what, when, why, where? Then how?– Data preservation

• Measurement methods used and the results obtained for the parameters of interest

• Uncertainties in the data that were assigned by the experimenters, how the uncertainties were determined and what they represent

8

ICSBEP Content and Format (Continued)

1.1 Overview of Experiment

• Summary of the experiment, its original purpose, the parameters that vary in a series of configurations, and mention of significant relationships to other ICSBEP-evaluated experiments

• Name of the facility, when the experiments were performed, the organization that performed the experiments, and perhaps the names of the experimenters if available

• The conclusions of the Evaluation of Experimental Data section, Section 2, should be briefly stated

9

ICSBEP Content and Format (Continued)

1.2 Description of Experimental Configuration

• Detailed description of the physical arrangement and dimensions of the experiment

– Discrepancies and inconsistencies should be noted

– Use original units (SI units then added parenthetically)

• Method of determining the critical condition and, if applicable, the measured reactivity are stated

• Subcritical measurements may require more detailed information about the source and detectors than is typically required for critical assemblies

10

ICSBEP Content and Format (Continued)

1.3 Description of Material Data

• Detailed description of all materials used in the experiment as well as significant materials in the surroundings

– Identify what information, if any, is missing– Note when density is calculated, and not reported

• Specify source of composition data (physical or chemical analyses or from material handbooks when only the type of material was specified

• Details of the methods of analysis and uncertainties• Dates of the experiment, of the chemical analysis, and

of isotopic analysis or purification– Usually when isotopic buildup and decay are important

11

ICSBEP Content and Format (Continued)

1.4 Temperature Data

• The temperature at which the experiments were performed should be given and discussed

• If available, how was the temperature measured

12

ICSBEP Content and Format (Continued)

1.5 Supplemental Experimental Measurements

• Additional experimental data (e.g., flux distributions, spectral indices, βeff, reactivity data, etc.) not necessarily relevant to the derivation of the benchmark model

• Subcritical measurements include a description of the measurement technology and a discussion on the interpretation of the measurements as well as the measured data

13

ICSBEP Content and Format (Continued)

2.0 EVALUATION OF EXPERIMENTAL DATA

• Evaluation of the experimental data and conclusions

• Missing data or weaknesses and inconsistencies in published data

• Effects of uncertainties

• Summary table

• Unacceptable data not included in Sections 3 & 4

• Unacceptable data may still be used in validation efforts if the uncertainty is properly taken into account

Section 2.0: Addressing Uncertainties

• Experimental Measurements

– Temperature• How does it impact

medium

– Worths

– keff

• Repeatability and reproducibility

– Control rod positions

• Geometrical Properties– Dimensions

– Quantity

– Position/location

• Compositional Variations

– Mass (density)

– Isotopic content

– Composition• Impurities

14

This section of the benchmark report has evolved significantly

since the initiation of the benchmark projects, due to the input

from many international experts over the past two decades.

Section 2.0: Addressing Uncertainties (Cont.)

• Computational Analysis– What is your statistical

uncertainty?• Needs to be larger than

your Δk calculations

– Perturbation Analysis• 1σ• Manufacturing tolerances• Repeated measurements• Measurement limits

– What is negligible?• Depends• Typically ≤0.00010 Δk

– What if statistical uncertainty is greater than calculated Δk?

• Perform longer calculations

– Still doesn’t work?• Assume linearity in Δk• Apply larger perturbation• Use a scaling factor

• Uncertainty Guide provided in Handbooks

– Examples in appendix– Also look at similar

evaluations

15

Benchmark Uncertainty vs. Criticality Safety Analysis

16

1σ = 0.1°C = negligible

Typical Probability Distributions

1. Normal: 3σ

2. Uniform, Equiprobable, Bounding: 1σ=Δx/√3

3. One-Sided Bounding, Triangular: 1σ=Δx/2√3

17

1.

2.3.

Probability Selection – Example I (HTTR)

18

Normal Distribution, 3σ

Probability Selection – Example II (NRAD)

19

1. Uniform,

Bounding2.

Bounding,

3σ or

Uniform?

Fabrication

Process?

3. ???Manufacturing Tolerances

What if the Uncertainty is Unknown?

• Example: pipe diameter

• What is the manufacturing tolerance?– National/International standards– Current fabrication capabilities

• Example: isotopic abundance of radioisotope

• What is the uncertainty in the reported values?– Typical uncertainties for similar experiments and

measurement capabilities– NIST traceable standards (systematic uncertainty)

• Does your uncertainty make sense?– Does is seem to adequately reflect reality?

20

Example – ORCEF HEU Metal (Y-12)

21

Not

always

available

Detection limit

What if You Have Too Many “Right” Answers?

• Example: composition of a material– Samples of a material distributed to different laboratories to

measure– Inter-laboratory results did not match within the reported

measurements and statistical uncertainties– Which is correct?

• Engineering judgment– Evaluate measurement capabilities and limitations– Can we redo the measurements?– Establish an nominal composition and uncertainty

• Does the uncertainty appear reasonable?• Does the uncertainty adequately encompass the measurement

uncertainty

• International review facilitates discussion of evaluation

22

What about Detection Limits?

• Bounding

• Do you expect a uniform distribution or triangular?– Was the material purified?

– Is the impurity typically not present?

23

Dealing with Unknown Systematic Uncertainties – i.e. Impurities

• What is a typical impurity content in a given material?

– Similar benchmarks, expert opinion, reference/manufacturing data, fabrication process details, comparison with similar materials

• Calculate effect in Δk for the inclusion of “typical” impurities in the material

• Take Δk/2 and apply as a bias for removal of unknown impurity content

– Include a bias uncertainty of ±Δk/2

24

Random vs. Systematic Uncertainties

• Random– Variation among

multiple measurements• Dimensions, masses, etc.

– Negligible for large sample populations

• TRISO and pebbles in a pebble-bed reactor

• Systematic– Biases in measurement

technique

– Not impacted by multiple measurements

• Impurities, enrichment, etc.

– Can be significant

25

Errors

(outliers)

Example – NRAD

26

•Bounding uncertainties

•Assumed grid hole

positions independent

•Assumed hole diameter

systematic due to single

drill bit and small

quantity of holes

•Use of URAN card

provided negligible

results (Δk ≤ 0.00010)

What is the Bottom Line?

• What is the “quality” of the experiment?

– Total uncertainty• How large is the

uncertainty?

– Large computational bias?• Due to missing or

erroneous data?

• Acceptable Benchmark Experiments

– For well-known materials• 1σ ≤ 1%

– For unique experiments, the uncertainty can be larger

– For large computational biases

• Are they within the uncertainty?

• Are they well known and quantified?

• Otherwise, unacceptable

27

Example – NRAD

28Δktot = √(Σ Δki

2) = 0.00271

29

ICSBEP Content and Format (Continued)

3.0 BENCHMARK SPECIFICATIONS

• Benchmark specifications provide the data necessary to construct calculational models

• Retain as much detail as necessary to preserve all important aspects of the actual experiment

• Simplifications include description of the transformation from the measured values to the benchmark-model values, the transformation and the uncertainties associated with the transformation

ICSBEP Content and Format (Continued)

3.1 Description of Model

• General description of main physical features of the benchmark model(s)

• Simplifications and approximations made to geometric configurations and/or material compositions are described and justified

• Resulting biases and additional uncertainties in keffare quantified

• Justification for omitting any constituents of the materials

30

Detailed vs. Simple Model(s)…What is the purpose?

31ZPPR-20 = SP-100 Mockup

32

Assessing Biases and Correction Factors

• The effect of simplifying the model is assessed by comparison of a detailed model to a simple model

• Some simplifications are anti-correlated– Their effects must be modeled individually and as a

whole to understand the complete result

• Sometimes the bias is smaller than the statistical uncertainty

– The bias is assumed negligible

– The bias uncertainty is included in the overall uncertainty of the benchmark model

33

Assessing Biases and Correction Factors (Cont.)

• Typical Examples– Measured worth of “critical” experiment

– Measured worth of components

– Room return effects

– Model Simplifications• Homogenization

• Adjustments for benchmark model temperature

• Geometry simplifications

• Removal of impurities

– What if true bias is unknown• Bias is estimated

• Bias uncertainty is increased

Example – GROTESQUE

34MCNP statistical uncertainty = 0.00003

35

ICSBEP Content and Format (Continued)

3.2 Dimensions

• Include all dimensions and information needed to completely describe the geometry of the benchmark model(s)

– Derived from Section 1; no rounding of units

• Sketches, including dimensions and labels, of the benchmark model(s) should always be included

36

ICSBEP Content and Format (Continued)

3.3 Material Data

• Atom densities for all materials specified for the model(s) are derived from the reported values given in the previous sections and are concisely listed

– Five significant digits

• Provide unique or complicated formulas for deriving atom densities

37

ICSBEP Content and Format (Continued)

3.4 Temperature Data

• Temperature data for the model(s)

38

ICSBEP Content and Format (Continued)

3.5 Experimental & Benchmark-Model keff and/or Subcritical Parameters

• Experimental values

• Benchmark values (adjusted to account for bias)

• Include total benchmark uncertainty – Combination of experimental & benchmark uncertainties

39

ICSBEP Content and Format (Continued)

4.0 RESULTS OF SAMPLE CALCULATIONS

• Calculated results obtained with the benchmark-model specification data given in Section 3

• Sample calculations only

• Discuss discrepancies between Benchmark values (Section 3.5) and calculated values (Section 4.0)

– C/E, (C-E)/E %, within 1σ or 3σ, biases, trends

40

ICSBEP Content and Format (Continued)

5.0 REFERENCES

• All formally published documents referenced in the evaluation that contain relevant information about the experiments

• References to handbooks, logbooks, code manuals, textbooks, personal communications with experts, etc. are given in footnotes

41

ICSBEP Content and Format (Continued)

APPENDICIES

• Supplemental information that is useful, but is not essential, to the derivation of the benchmark specification

– Calculations, photos, scanned documentation, model variants, auxiliary measurements

APPENDIX A

• Typical Input listings

• Brief comments about options chosen for calculations

• Version of the code (e.g., MCNP, KENO, MONK, etc.), SN Codes: quadrature order, scattering order for cross sections, Convergence criteria, representative mesh size, Monte Carlo Codes: number of active generations, number of skipped generations, total number of histories, etc.

42

IRPhEP Evaluation Format

• Experiment title

• Identification number(Reactor Name) - (Reactor Type) - (Facility Type) - (Three-Digit Numerical Identifier)

(Measurement Type(s))

Reactor Type Facility Type Measurement Type

Pressurized Water Reactor PWR Experimental Facility EXP Critical Configuration CRIT

VVER Reactors VVER Power Reactor POWER Subcritical Configuration SUB

Boiling Water Reactor BWR Research Reactor RESR Buckling & Extrapolation Length BUCK

Liquid Metal Fast Reactor LMFR Spectral Characteristics SPEC

Gas Cooled (Thermal) Reactor GCR Reactivity Effects REAC

Gas Cooled (Fast) Reactor GCFR Reactivity Coefficients COEF

Light Water Moderated Reactor LWR Kinetics Measurements KIN

Heavy Water Moderated Reactor HWR Reaction-Rate Distributions RRATE

Molten Salt Reactor MSR Power Distributions POWDIS

RBMK Reactor RBMK Nuclide Composition ISO

Fundamental FUND Other Miscellaneous Types of Measurements

MISC

43

IRPhEP Evaluation Format (continued)

1.1 Description of Critical and Subcritical Measurements

1.2 Description of Buckling and Extrapolation Length Measurements

1.3 Description of Spectral Characteristics Measurements

1.4 Description of Reactivity Effects Measurements

1.5 Description of Reactivity Coefficient Measurements

1.6 Description of Kinetics Measurements

1.7 Description of Reaction Rate Distribution Measurements

1.8 Description of Power Distribution Measurements

1.9 Description of Isotopic Measurements

1.10 Description of Other Miscellaneous Types of Measurements

Summary of IRPhEP Measurements

44

Additional Benchmark Measurements

• Measurement uncertainty typically dominates uncertainties in geometry and materials

• What about βeff?– Difficult to measure

– Variation between different cross section data libraries typically large anyway

• How well do measurement techniques/methods compare to computational simulations

45

Questions?

46

HTR-PROTEUS