G. Cowan, RHUL Physics Alternative to CLs page 1

Proposal for alternative to CLs:Power-Constrained Limits

ATLAS Statistics Forum

CERN, 25 May, 2010

Glen Cowan, RHULKyle Cranmer, NYUEilam Gross, Ofer Vitells, Weizmann Inst.

G. Cowan, RHUL Physics Alternative to CLs page 2

The “CLs” issue

When the cross section for the signal process becomes small (e.g., large Higgs mass), the distribution of the test variable used in a search becomes the same under both the b and s+b hypotheses:

In such a case we will reject the signal hypothesis with aprobability approaching = 1 – CL (i.e. 5%) assuming no signal.

f (q| b)

f (q| s+b)

G. Cowan, RHUL Physics Alternative to CLs page 3

The CLs solutionThe CLs solution (A. Read et al.) is to base the test not onthe usual p-value (CLs+b), but rather to divide this by CLb (one minus the background of the b-only hypothesis, i.e.,

Define:

Reject signal hypothesis if: Reduces “effective” p-value when the two

distributions become close (prevents exclusion if sensitivity is low).

f (q| b)

q

f (q| s+b)

G. Cowan, RHUL Physics Alternative to CLs page 4

Alternative proposal – basic ideaCLs method reduces the p-value according to:

where = strength parameter, proportional to cross section.

Statistics community does not smile upon ratio of p-values;would prefer to regard parameter as excluded if:

(a) p-value of < 0.05(b) power of test of with respect to background-only > some threshold

Requiring (a) alone gives the standard frequentist interval, (CLs+b method) which has the correct coverage.

Requiring ANDed combination of (a) and (b) is more conservative;end effect is similar to CLs, but makes more explicit the minimumthe role of minimum sensitivity (as quantified by power).

G. Cowan, RHUL Physics Alternative to CLs page 5

Similar to….

Feldman and Cousins touched on the same idea in connectionwith FC limits:

We propose to make this more explicit using the power of thetest of a given strength parameter m with respect to thealternative background-only hypothesis.

G. Cowan, RHUL Physics Alternative to CLs page 6

Formalizing the problem

In the context of tests based on the likelihood ratio (),the p-value can be written

The upper limit is found by setting p = and solving for ,

Estimator for strength parameter

Standard normal cumulative dist.

G. Cowan, RHUL Physics Alternative to CLs page 7

False exclusion rate for no sensitivityExcluding if p < gives right coverage, but this meansthat probability to exclude in case of no sensitivity is .

To see this note, probability to exclude assuming = 0 is

“No sensitivity” means / « . In this limit, the false exclusionprobability becomes

G. Cowan, RHUL Physics Alternative to CLs page 8

Power of test of relative to = 0The power of a test of relative to the alternative = 0 is

or equivalently in terms of the distribution of up,

G. Cowan, RHUL Physics Alternative to CLs page 9

Criterion for rejecting We formulate the criterion for rejecting a hypothesized byRequiring p < and also that the power be greater than aminimum threshold 1 – ′. (i.e. Type-II error rate < ′ ). Thepower-constrained limit is thus

where ′ is the for which the power is ′.

The requirement implies sothe minimum power requirement can be expressed

or equivalently

G. Cowan, RHUL Physics Alternative to CLs page 10

Choice of minimum power

Note that if the minimum power 1 – ′ = (typically 0.05), then ′ = 0, and then pc = up always.

Normally would choose < 1 – ′ ≤ 0.5. Convention must bediscussed (also with CMS).

Coverage of power-constrained interval is well defined:

95% for pc = up 100% for pc < up

G. Cowan, RHUL Physics Alternative to CLs page 11

Solution in terms of median p-valueBecause of the monotonic relation between the p-valueand estimator for , the median of pm assuming = 0 is:

In addition to the median (50% quantile) we can also find thequantiles corresponding to the N deviations of muHat:

G. Cowan, RHUL Physics Alternative to CLs page 12

Extra slides

G. Cowan, RHUL Physics Alternative to CLs page 13

A possible experimental outcomeSuppose a given experiment gave the following p-value versus :

Here data have clearly fluctuation low.

G. Cowan, RHUL Physics Alternative to CLs page 14

Choice of likelihood ratio statistic

Ongoing discussion as to whether best to use LEP-stylelikleihood ratio

or

and in both cases how to deal with the nuisance parameters.

In simple cases one obtains the same test from both statistics.