Implementation: Pilot testing – and further development – a critical next step

David F. Ransohoff MDDepartments of Medicine and EpidemiologyUniversity of North Carolina at Chapel Hill

Implementation steps: •certify organizations

from p146,7

Implementation steps: •certify organizations •what’s missing: how to measure

“how trustworthy” is individual CPG

from p146,7

How trustworthy is a CPG?

a. How is “trustworthy” measured (scale) •existential (yes/no)•ordinal (very trustworthy, moderately..., not)•numerical

b. How are the 8 “standards”:•measured individually •‘combined,’ to decide (a) “how trustworthy”?

Doing (a) and (b) is ‘scale-making’

Scale-making is commonly done e.g, Apgar Score, or SF-36 These scales “components” have multiple categories. For many scales “coding manuals” are developed.

How trustworthy is a CPG?Example: What happens if we apply “standards” to CRC screening guidelines,

that were a “call-out”: can we tell how trustworthy each is?

How trustworthy is a CPG?(CRC MSTF 2008)

... according to a panel chair:“[T]he process ultimately became politicized and, according to one participant, resembled “sausage-making.”’

-Cancer Letter, Oct 12, 2008

How trustworthy is a CPG?

Current Instructions to apply standards

How trustworthy is a CPG?

Current Instructions to apply standards

Current instructions ask only: “Is everything perfect.” There’s no “threshold,”

and no details for individual standards. “Pilot testing” could show whatwould user (CMS? BC/BS? Kaiser?) think or do; how would they use?

And then are YOU (IOM) satisfied with how they use them? Do current standards solve the problem we’ve got in this field?

Why might we need a more-explicit way to measure “how trustworthy”?

1. Even if you “certify organizations,” must also be able to judge organizations’ products. (e.g., pharmaceuticals)

2. Many guidelines will be made by “uncertified organizations” (or do you expect all to stop?).

3. So a user then needs to know how to judge a CPG – individual elements, ‘composite score’.

4. #3 was IOM’s original charge.5. If this were a pharmaceutical, would it be “rolled out”

without #3, and without testing? (Does this solve problem in field?)

6. We need strong checks-and-balances in an environment when it’s hard for professionals to be “professional.”

What a “profession is (via Louis Brandeis and Sam Thier)

Why Sam Thier talked about this all the time: ‘Society grants privileges to professions (title, income)... in exchange for ‘pursued largely for others’. If a profession doesn’t act right, privileges will be taken away.

Example of one profession’s concerns re future (AGA Institute Future Trends Committee conference, 2006)

Example of one profession’s concerns re future (AGA Institute Future Trends Committee conference, 2006)

‘The Doomsday Conference’

Separate example: May 9, 2011 AGA Plenary, on screening: “Disruptive Technology”

Checks and balances

1. “Professional organization” wears 2 conflicting hats:1) clients’ interest comes first (Brandeis; Thier)2) professionals’ economic interest comes first

2. Intellectual conflict exists apart from economics: people “believe in” what they do. (Fowler FJ, et al. Comparison of recommendations by urologists and radiation oncologists for treatment of clinically localized prostate cancer. JAMA 2000;283:3217)

Checks and balances

3. Lessons from evolution of “composition” of ACS-MSTF (Multi Society Task Force)•# generalists/methodologists in MSTF author group

1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594

Checks and balances

3. Lessons from evolution of “composition” of ACS-MSTF (Multi Society Task Force) •# generalists/methodologists in MSTF author group

1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594

Why, btw, was MSTF formed? (Sid W; AGA; May 9)

Checks and balances

3. Lessons from evolution of “composition” of ACS-MSTF (Multi Society Task Force)•# generalists/methodologists in MSTF author group

1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594

2003: 2 (RHF, SHW) Gastroenterology 2003;124:544

2008: 0 Gastroenterology 2008;134:1570

Checks and balances3. Lessons from evolution of “composition” of

ACS-MSTF (Multi Society Task Force)•# generalists/methodologists in MSTF author group

1997: 4 (RHF, FG, CDM, SHW) Gastroenterology 1997;112:594

2003: 2 (RHF, SHW) Gastroenterology 2003;124:544

2008: 0 Gastroenterology 2008;134:1570

The Task Force was “multisociety”; but over time, dominated by subspecialists and lost all generalists/methodologists.

Each iteration of the guidelines was “less trustworthy”; by2008 “very not trustworthy”.

Evolution reflect “inclination”, and power of economic hat.Anecdotes about guidelines-making are ugly, disturbing, and

cleverly hidden. Not ‘good faith’.

Checks and balances

The challengeGuidelines that are in clients’ interests:•really need the expertise hat•really need to avoid the economic hat.

Do current standards deal strongly-enough with this tension? I think not.

Conclusion and Suggestion

1. IOM (someone) needs to further develop details of standards and of implementation, to provide effective checks-and-balances.

2. Jefferson and Madison didn’t do their work “suddenly.” It required time/experience to develop checks and balances. Can we establish “process”, mechanism to test, develop standards, “trustworthy” and see if they look like they are solving problem? Who does this?

3. Guidelines-making is a central issue in “professionalism” in our era; a neutral, professional, authoritative group (IOM?) needs to do/direct this. Can anyone else?