BOOTSTRAPPING& DISSOLUTION SIMILARITY: Q&A

Jiri Hofmann

BioBridges 2019

Prague, September 26-27, 2019

SIMILARITY: CLAIM OF EQUIVALENCE

reference

test

Time (hours)

% r

ele

ased

Test Reference

Test A Test B

Biobatch Biobatch

Pilot Clinical

Strength 1 Strength 2

API 1 API 2

+Excipient −Excipient

Site 1 Site 2

…

Type

(M)ANOVA

f1(2)

Bayesian

MD

Bootstrap

MSD

T2 test

EDNE test

PCA

…

(M)ANOVA (multivariate) analysis of variance; f1(2) similarity factor; MD maximum

deviation; MSD multivariate statistical distance; EDNE Euclidean distance of the

nonstandardized expected (values); PCA principal component analysis

FIT FACTOR: MOORE & FLANNER

Sim

ilari

ty f

acto

r

Time (min)

% r

ele

as

ed

reference

test

Mean difference (%)

−10%

𝑓2 = 50 𝑙𝑜𝑔 1 +1

𝑛

𝑖=1

𝑛

𝑹𝒕 − 𝑻𝒕2

−0.5

× 100

Moore & Flanner (1996). Pharm Technol 20(6): 64-74

𝒇𝟐 = 𝟒𝟗. 𝟖𝟗𝟏𝟗𝟕

Time (min)

% r

ele

ased

reference

test

+3.7%

−0.4%

−6.0%

−10.7%

−13.7%

−15.4%

𝒇𝟐 = 𝟓𝟎. 𝟏𝟎

9.95%

similar

dissimilar

−10%−10%

−10%−10%

SIMILARITY FACTOR: WHAT‘S WRONG WITH YOU?

Time (min)

% r

ele

ased

𝒇𝟐 (𝟏−𝟔) = 𝟓𝟏. 𝟖𝟓

Time (min)

reference

test

Time (min)

Shape & time correlationVariabilityNumber of points

1

3

2

45 6

𝒇𝟐 (𝟏−𝟒) = 𝟒𝟕. 𝟖𝟗

3

5

2

64

1

𝒇𝟐 (𝟏−𝟔) = 𝟓𝟏. 𝟖𝟓

85%

𝑓2 = 50 𝑙𝑜𝑔 1 +1

𝑛

𝑖=1

𝑛

𝑹𝒕 − 𝑻𝒕2

−0.5

× 100

(…) impossible to evaluate false

positive or false negative (…). (…) too

liberal in concluding similarity.

Liu et al. (1997). Drug Info J. 31: 1255-1271

(…) f2 (…) series of monotone (…)

transformations of the Euclidean

distance, (…) procedure where neither

the type I error rate nor power can be

controlled and which really hurts a

statistician‘s soul.

Hoffelder (2019). Biom J. 61(5): 1120-1137

STATISTICIANS: VIEW ON SIMILARITY FACTOR

Cartesian coordinate system

𝒅(𝒑, 𝒒)

𝑨

𝑩2

3

𝒑𝟏 𝒒𝟏

𝒑𝟐

𝒒𝟐

x

y

EMA: REFLECTIONS

[6.3] The f2 (…) unfavorable statistical

properties (…), no (…) quantification

of the risk to false positively conclude

on similar dissolution is possible.

[Answer:] (…) is based only upon the

(…) numerical value for f2 (point

estimate ≥ 𝟓𝟎). (…) uncertainty related

to the f2 sampling distribution is not

accounted for.

23 March 2017

EMA/CHMP/138502/2017

Committee for Human Medicinal Products (CHMP)

Reflection paper on statisticalmethodology for the comparativeassessment of quality attributes indrug development.

26 July 2018

EMA/810713/2017

Human Medicines Research and Development Support

Question and answer on the adequacyof the Mahalanobis distance to assessthe comparability of drug dissolutionprofiles

The aim (…) to a larger extent to emphasise the importanceof confidence intervals to quantify the uncertainty aroundthe point estimate of the chosen metric (…).

1

2

3

4

5

6

7

DISSOLUTION SIMULATION: 𝑦 = 100 × 1 − 𝑒−𝑘𝑡

Population 𝒇𝟐

Sim

ilari

ty (

%)

Time (min)

% r

ele

ased

reference

test

Population 𝒇𝟐 = 𝟒𝟎

𝒌 = 𝟎. 𝟎𝟔𝟎𝟎

𝒌 = 𝟎. 𝟎𝟑𝟓𝟔

Empirical power

𝒇𝟐 = 𝟑𝟖

𝒇𝟐 = 𝟒𝟐

1

N

N=10‘000

Samples

𝒇𝟐 > 𝟓𝟎N

𝜶 = 𝟓%

Time (min)

% r

ele

ased

𝒌

ෝ𝜶 > 𝟓𝟎%

CV

(%

)

Time (min)

𝒇𝟐 = 𝟒𝟎

1

6

𝝈𝟐1

6

Mean variability

12

12

% r

ele

ased

12

12

EMA Q&A: BOOTSTRAP

[Answer:] (…) bootstrap methodology

could be used to derive confidence

intervals for f2 (…), (…) the preferred

method over f2 and MD. EMA/810713/2017

boot·strap /ˡbu:tstræp/ noun IDM pull /

drag yourself up by your (own)

bootstraps (informal) to improve your

situation yourself, without help from

other people

Oxford Advanced Learner‘s Dictionary of Current English

(2000). Oxford University PressRaspe (1785): Baron Munchausen's

Narrative of his Marvellous Travels

and Campaigns in Russia

3.08 4.92

BOOTSTRAPPING: PRINCIPLEUnknown population of cannon balls

5

4

1

Original data 𝒙 N=12

N=12

N=12N=12

7

6

2 32

6

43

1

7

6

5

41

3

2 𝑥∗2

𝑥∗𝐵2

6

6

3

4

7

7

ҧ𝑥 = 4.00

ҧ𝑥∗1 = 3.17

ҧ𝑥∗2 = 3.92 ҧ𝑥∗𝐵 = 4.00

𝑥∗1

Fre

qu

en

cy

Bootstrap distribution

𝒔𝒆𝑩 = 𝟎. 𝟓𝟔

sample

sample

sample

90%CI

B=10‘000

𝒔𝒆 =𝒔𝒅𝒙

𝑵= 𝟎. 𝟓𝟗

𝒔𝒆𝑩 bootstrap standard error

𝒔𝒆 standard error of the mean

5% 95%

3.173.92...

4.00

bootstrap replicates

bootstrap sample

BOOTSTRAPPING DISSOLUTION: CI

90% bootstrap CI(1)

Type Lower Upper

Normal 60.2217 66.9493

Percentile 59.8591 66.5973

Basic 60.5737 67.3119

BC 60.6481 67.4051

BCa 60.7555 67.5536

Bootstrap-t(2) 60.5844 67.8099(1)R (v3.6.1); B0=10‘000; (2)B1=1‘000; BC

bias-corrected; BCa bias-corrected and

accelerated.

𝒇𝟐∗𝑩 = 𝟔𝟏

𝒇𝟐∗𝟏 = 𝟔𝟒

𝒇𝟐∗𝟐 = 𝟔𝟎

1

2

B

𝒇𝟐∗𝒊

sample

sample

reference

test

Time (hours)

N=12

N=12

Similarity (...) be declared if the

CI for f2 (…) entirely above 50.EMA/810713/2017

12

12

12

12

12

12

𝒇𝟐𝒙 = 𝟔𝟒

BOOTSTRAPPING DISSOLUTION: ALL EASY?

90% bootstrap CI(1) (data I)

Type Lower Upper

Normal 75.3720 105.6605

Percentile 62.7877 092.9877

Basic 88.0448 118.2448

BC 87.4104 099.7090

BCa 87.4081 099.7060

Bootstrap-t(2) 87.9071 123.3391(1)R (v3.6.1); B0=10‘000; (2)B1=1‘000

90% bootstrap CI(1) (data II)

Type Lower Upper

Normal 49.0571 65.8814

Percentile 50.5090 67.4505

Basic 47.4880 64.4295

BC 50.4077 67.3106

BCa 49.9249 66.4951

Bootstrap-t(2) 47.5351 66.7653(1)R (v3.6.1); B0=10‘000; (2)B1=1‘000

BOOTSTRAPPING DISSOLUTION: ALPHA & POWER

Population 𝒇𝟐

Sim

ilari

ty (

%)

𝒇𝟐 point estimate

5% percentile

Low %CV in dissolution High %CV in dissolution

Population 𝒇𝟐

𝒇𝟐 point estimate

5% percentile

N=10‘000

B=1‘000

CV

(%

)

CV

(%

)

Time (min) Time (min)

𝒇𝟐 = 𝟏𝟎𝟎 𝒇𝟐 = 𝟏𝟎𝟎

1

6

1

6

ෝ𝜶 > 𝟓% ෝ𝜶 > 𝟓%

Sim

ilari

ty (

%)

[Major objection] Biowaiver (…) may not

be sufficiently supported (…). (…) 90%

confidence interval of the f2 similarity

factor calculated by bootstrapping

(…). The output should include at

least the number of bootstraps, the

5%, 50% (median) and 95% percentiles.

Evidence that the statistical software

has been validated should be

provided as well.

BOOTSTRAP IN PRACTICE: DL

Percentiles (P)

P Shah et al. DDSolver(1) RE (%)

5% 53.01 53.53147 −0.97

95% 68.34 68.72088 −0.55

B=1‘000; (1)Zhang et al. (2010)

reference

batch 1

Shah et al., 1998

Time (hours)

% r

ele

ased

Shah et al. (1998). Pharm Res 15(6): 889-896;

Zhang et al. (2010). AAPS J. 12(3): 263-267

input

output

software

BOOTSTRAP IN PRACTICE: ROUND 1

Percentiles

05% 86.9786

50% 89.3210

95% 91.5900

B=50‘000 in DDSolver

Percentiles

05% 50.6544

50% 52.8409

95% 55.7312

B=50‘000 in DDSolver

biobatch

waiver

Time (min)

% r

ele

as

ed

AGJ pH 1.2 Phosphate buffer pH 4.5 Phosphate buffer pH 6.8

0.00068 mg/mL 0.0013 mg/mL 0.11 mg/mL

Percentiles

05% 97.9487

50% 98.9720

95% 99.5367

B=50‘000 in DDSolver

biobatch

waiver

biobatch

waiver

[Response to draft] (…) to construct

more versions of 90% confidence

interval (…) to see if results regarding

similarity are sufficiently robust.

BOOTSTRAP IN PRACTICE: ROUND 290% bootstrap CI (Bootf2BCA)(1)

Type Lower Upper

Percentile 50.6599 55.7327

Bootstrap-t(2) 50.7184 56.6114

Normal 50.3144 55.4415

BCa 50.9257 56.3030

Basic 50.1124 55.1852(1)Mendyk et al., 2013; (2)in-house R-code,

R (v3.6.1); B0=50‘000; B1=1‘000

biobatch (1 tablet)

waiver (4 tablets)

Time (min)

% r

ele

ased biobatch (1 tablet)

waiver (1 tablet)

% r

ele

ased

Mendyk et al. (2013). Diss Technol 20(1): 13-17

Phosphate buffer pH 6.8

Dose 𝑨 = 𝑩

+0.5% Tween 20

[Response to draft] The assessor still finds

unclear what is difference between

normal approximation method and

basic bootstrap (…). The Applicant is

asked to fill in this information (…).

[Response to final] (…) the Applicant is

asked to provide (…) software

validation for individual approaches to

see that obtained results can be

considered as relevant.

BOOTSTRAP IN PRACTICE: ROUND 3 & 4

reference

test

Time (hours)%

rele

ased

𝜃 − 𝑏𝑅 ∓ 𝑣𝑅Τ1 2 ∙ 𝑧 1−𝛼

2 𝜃 − 𝜃∗ 1−𝛼 , 2 𝜃 − 𝜃∗ 𝛼

Davison & Hinkley (1997). Cambridge Press

Normal approximation

Basic (backwards)

SUMMARY & QUESTIONS

● f2: no control of type I error Statistics vs. clinical relevance? Signalsof problems available?

● Bootstrap f2: confidence interval … but

harder to pass similarity criterion

Type I error control? Which CI?

Comparison to other methods?

Applicable generally (EMA Q&A)?

● Equivalence margin: average 10% Current margin already conservative?Maximum 10% (EMA Q&A) or average10%?

● Dissolution in non-sink conditions API or product performance?

> library(fortunes)

> fortunes::fortune(222)

Some people familiar with R describe

it as a supercharged version of

Microsoft's Excel spreadsheet

software.

-- Ashlee Vance (in his article "Data Analysts Captivated

by R's Power") The New York Times (January 2009)

> library(fortunes)

> fortunes::fortune(358)

The existence of a method is not a

sufficient reason to use that method.

-- Jari Oksanen (about relative advantages of several

multivariate analysis methods) R-SIG-Ecology

(November 2013)

QUOTES: LIBRARY(FORTUNES)

R Core Team (2019). https://www.R-project.org/