2012 Inaugural “Digital Pathology Showcase”

Draft – Version 3 for Public Comment

ObjectiveThe primary goals of this event are to provide the audience with a good sense of the state-of-the-art in WSI-based digital pathology and provide visibility into the viability and readiness of digital WSI technology for routine use in pathology: “is it ready for prime time?” To accomplish that, a set of pathology cases will be provided by the organizers and reviewed live by practicing pathologists. Participating vendors will provide their equipment to the pathologists to showcase the capabilities of their WSI technology. Systems will be subjectively and informally judged by the audience, observing the time it takes and the quality of the diagnostic recommendation based on the WSI reviews.

The inaugural event will exclude issues such as workflow optimizations via LIS integration and advanced Computer Assisted Diagnostics (CAD). The case reviews will be done based on manual review of the WSI and using basic annotation capabilities. In future years, we do expect this to grow and include some of the excluded aspects.

LogisticsWe are anticipating 3 vendors participating in the 2012 event with duration of approximately 3 hours. The order of presentation will be pre-determined so that each vendor will present first, second and third of the three scenarios proposed. We plan to interleave the review of the three types of pathology cases so that a subjective side-by-side comparison will be possible for the audience. We are considering some overlap of the scanning for the second case scenario (multi slide biopsy with a focal cancer) to ensure the even can be conducted in a timely manner. We will make that determination once we do have confirmation of the three participating vendors.

CasesEach vendor will be given a set of representative cases in advance of the event so that it is known what type of cases will be expected to be reviewed during the event. Except for the first case, those cases will be replaced prior to the event to ensure the pathologists do not know the diagnostic outcome of the case but will have to perform a “live” review. The three sets of cases will be:

1. Frozen section: 1 slideSlides will be dry and scanned as well as reviewed “live”. The total time to scan, transfer, and diagnose the slides “live” will be observed by the audience.

2. Biopsy with a focal cancer: 6 to 9 slides

Case will consist of 2 or 3 blocks and 3 slides per block, H&E only, various cases of infiltrating breast. Prostate or colon carcinomas with the cancer being present it at least one of the blocks. The case shared ahead of the event and the one being reviewed during the event will not be the same. New slides will be provided shortly prior to the session so that there is enough time to scan but no access to the images by the pathologist, thus ensuring a “live” review of the case. The time it takes the pathologist to come to their diagnosis will be observed by the audience.

3. Immunohistochemical Quantitative Interpretation: 1 slideCase will consist of one immunohistochemical stain of a breast biopsy for ER, PR or Her2Neu interpretation. The case shared ahead of the event and the one being reviewed during the event will not be the same. New slides will be provided shortly prior to the session so that there is enough time to scan but no access to the images by the pathologists, thus ensuring a “live” review to determine the finding in terms of the biomarker examined..

RulesThe organizers of the event are setting forth a set of rules to make this a valuable comparison. In the following we list the guidance for the vendors on what is expected as well as permitted:

Demonstration setup is required to include scanner as well as needed workstation and server(s) Demonstration to be conducted by practicing pathologist(s) and a pathology technician

(histotechnologist, IT or scanner personnel or other non-vendor personnel. No vendor employees are allowed to take place in the actual demonstrations.

Software needs to be a commercially released version, no Alpha or Beta software allowed. Hardware equipment needs to reflect a common commercial deployment (e.g. do not replace

standard hard disks with solid state drives to boost performance beyond what customers can afford in a real deployment)

Prefetching or pre-caching of cases will not be allowed and network utilization will be monitored

Additional rules might need to be put in place as we refine the draft of this document.

SetupThe setup will include scanners, client workstations, and possibly server infrastructure, depending on the deployment needs of each vendor. Vendors are asked to setup the equipment the evening before the event and will be assisted with respect to the network setup.

The network bandwidth between the scanners and the respective storage unit (server) will vary based on each case review. For the Frozen section, a 2Mbit/sec network bandwidth and 100ms network latency will be applied while for the other two cases 20Mbit/sec network bandwidth and 30ms network latency will be applied. This is to reflect reasonable network conditions for a remote read of a frozen section as well as regular reads throughout a hospital network. If selected vendor(s) requires a higher

degree of network performance that will be considered and all vendors will have access to any upgrades from this specification that occur.

A total of three projection screens will be available. Screen #1 will display a view of the respective scanner during the “live” scanning of the frozen section scanning and subsequently a view of the keyboard + mouse of the pathologist reviewing each case. Screen #2 and #3 will show a view of the two screens of the workstation the pathologist will use to read a case. This setup will allow the audience to fully comprehend the effort and assess ease of use of reviewing each case.

Appendix

Suggested room layout: (This can be altered to fit the available space)

Suggested network setup:

Comments from Vendor Interviews (Draft from MB – some not approved/verified for release)

Interviews with Industry Partners

Mark Newberger (MN), Apollo Telemedicine, Inc June 6th, 2011Interviewer – Mike Becich (MB)

Summary of Mark’s Newberger’s comments:1) Overall MN thought the idea of a digital pathology showcase as proposed in 4/20 Draft is a good

idea2) He also thought the 18-20 month lead time for the showcase in fall 2012 was a good idea as well3) He did admit that he initially thought we were planning to have the showcase this fall with a 4 to

6 month lead time4) He had issues with companies like Apollo (which does workflow software and is scanner

agnostic) would have to partner with one or two other companies to compete in the showcase.5) He didn’t feel that the current draft would yield more than two or three vendors who could do

end to end (scanning, processing/display and diagnostic workflow) digital pathology6) He favored breaking the showcase into segments (scanning, processing/display and diagnostic

workflow) and having different vendors for each of the segments7) MN suggested that having a “training” program for the vendors on the technical set up a day

before the 2012 meeting was a good idea8) MN and MB suggested that the technical specification for the showcase set up be made

available for testing at the company sites. MB suggested that Pittsburgh would offer a “test” setup in Pittsburgh at least 11 months before the 2012 meeting to let companies decide if they want to participate (test cases and technical set up).

9) MN agreed that offering the first six vendors who committed to the showcase was a good idea but was concerned you might not get six the first year. Vendors might be concerned how they would “show” in this type of open showcase.

10) MN also thought the showcase would provide good marketing potential to their customers as well as new customers.

11) MN agreed the “open” sharing of the whitepaper and the forum discussion as an industry panel at Pathology Informatics 2011 was a good idea.

12) MB closed the meeting by asking MN if he would mind sharing his comments on the showcase with the broad pathology and vendor community and he agreed.

Interviews with Industry Partners

Ole Eichhorn (OE), Aperio June 10th, 2011Interviewer – Mike Becich (MB)

Summary of Ole Eichhorn’s comments:1) Overall OE thought the timing and planning for the industry panel in 2011 and the digital

pathology showcase in 2012 was a good plan2) OE stated that Aperio considered being involved in the European Scanner Contest in 2010 but

decided not to participate because of the “closed” nature of the planning and evaluation3) OE felt the current plan for the 2012 Digital Pathology Showcase was a starting point but

suggested more complex activities (e.g. quantitative analysis and algorithms) might be considered as well.

4) OE stated that Aperio (as leader in the field) was interested in further dialogue on the format of the showcase in 2012 and would circulate the document to others in Aperio..

Interviews with Industry Partners

John Wellbank (JW) and Jose Costanon (JC), PhillipsJune 17th, 2011Interviewer – Mike Becich (MB)

Summary of John Wellbank’s and Jose Costanon’s comments:1) Both JW and JC liked the idea of the industry panel in 2011 and the digital pathology showcase

in 2012 but raised some concerns.2) JC felt that Phillip’s would like to see higher volume of slides scanned included in the showcase.

When MB told them that there were plans for 10 to 20 slides for the second (biopsy) and third (resection) case this did not fit the “higher volume” scenario. MB assured JC that this would be considered and discussed in the industry panel in 2011 and as part of the “open” comments from the community. JW added that Phillips would need scanning in 20 and 40x as well as more volume if possible. Phillips does not scan in 20x equivalency so that test is not possible. Therefore we need at least some of this test to scan resolutions equivalent to 40x for true time comparisons

3) JW was concerned that the network bandwidth and latency listed in the current (attached) draft of the plan were not sufficient from Phillip’s perspective. MB reassured both JC and JW that network bandwidth and latency will be provided to the highest specification requested by the industry partners. MB asked JW/JC to provide a network specification and that MB would make sure this was part of the plan.

4) JW felt the bar might be set “too low” for the first meeting and would provide additional feedback from others at Phillips on the case scenarios.

5) JW was concerned that there would be no time for vendors to test the setup for the Industry Showcase. MB reassured JW that a “pilot setup” (simulating the setup for 2012) would be available in Pittsburgh for at least 11 months in advance of the showcase and made available for scheduling to industry partners. MB also stated that the specification for the Industry Showcase would be made available to all interested industry partners in November 2011 should an industry partner want to replicate the environment in the company headquarters.

6) Overall JW and JC thought that the timing (over 18 months of lead time) and consensus building with industry partners would ensure a successful showcase.

Interviews with Industry Partners

Philippe Nore (PN), Ventana July 5th, 2011Interviewers – Mike Becich (MB) and Bruce Friedman (BF)

Summary of Philipe Nore’s comments:1) Don’t want a contest that “compares” vendor on only one dimension (perhaps speed of

scanning or image quality).2) Don’t want to promote instruments that are “skewed” on the basis of any one factor (e.g. a

“contest” specific set of criteria).3) Don’t want to end up pushing company production of instruments and software to “win the

contest”. The focus should be on how to meet the needs of customers.4) Question – Will image analysis be a part of the first showcase? Ventana would like to see a IHC

or ISH image analysis as part of the first showcase. PN felt many of the initial digital pathology solutions use this today, so why not showcase this to customers today.

5) Will the event will have time for six vendors demos (may be too many).6) PN thought that even having 4 vendors would be hard to keep audience’s attention through the

entire event and should limit time for each exercise.7) Ideally for any event you would want to be vendor #1 or 2 but not 3 or 4 (much less 5 or 6) in

the current setup due to “repetitive/redundant” nature of the tasks8) Commentary – reduce the time for each serial demo by vendor.9) PN thought by 2012 everyone would have a image analysis algorithm.10) PN thought 20 minutes per vendor should be considered – based on his experience with

customers.11) PN asked what “live mode” means. Does this mean scanning occurs on stage. In the Ventana

system the slide viewing is dynamic robotic so scanning a frozen section is not required once mounted on their hardware. Ventana would like the flexibility to view the slide without scanning for the frozen section exercise.

12) No FDA approval for primary diagnostics so exercise # 2 and # 3 may be out of question. They need to check with their regulatory lawyers to make sure this event is misperceived as promoting non-FDA approved use of devices.

13) User training is an issue and need this to be addressed. Who will run the demos? How will Pathology Informatics deal with this – told PN about the 3 solutions we propose for this including:

a. Availability of a “mock” setup in Pittsburghb. Availability of a “specification” for the setup and encouraging replication of the setup at

the vendors headquarters or at a key clients location.c. Training experience on the “mock” system in Pittsburgh will be shared with

users/vendors.14) Ventana would require their own scanner, image server and application server loaded in

advance. Can’t share with others. Will follow up with PN on the specs.15) Separate fee for vendor participation in the Industry Showcase was PN’s recommendation.

Rob Liepse (RL) and Stephane Willaert (SW), Barco July 20th, 2012Interviewer – Mike Becich (MB)

Summary of Rob Liepse’s and Stephane’s Willaert (SW) comments:1) SW commented that the European contest was only focused on the scanner. SW felt that the “end

to end” solution including software, servers, etc... to be included in the showcase was a good focus for the 2012 event.

2) SW asked if there would be opportunities to have multiple vendors partner for the various partners for the component technologies.

3) SW questioned what role did I envision for Barco in the Industry Showcase and it seemed Barco’s role would be in the display of scanned slides.

4) SW stated in the Berlin scanner contest their role was as the display partner for all the vendors that participated.

5) SW stated that Barco is working with all the vendors in the digital pathology space.6) RL mentioned that they have a very large footprints in digital mammography and PACS displays.

They have been a repeat vendor at RSNA for years and participated in IHE (Integrating Healthcare Enterprise). They have extensive experience in primary diagnosis in the mammography space.

7) RL mentioned that Barco, worlds only approved display for TOMO (tomosynthesis – mammography tomography – have 510K approval) and have been very involved in standards efforts and moving this technology to the masses.

8) SW thought that the focus of the proposed Industry Showcase was appropriate but that it would make a number of the vendors nervous due to the lack of industry focus on “end to end” solutions. SW felt this would move the industry forward but it may be ambitious.

9) SW thought the technical focus on bandwidth support for the demo was a good one and that not enough studies are looking at the effect of the network on the viewing experience of the diagnosing pathologist.

Interviews with Industry Partners

Doug Giszczynski (DG) and Donal O’Shea (DO), LeicaJuly 26th, 2011Interviewer – Mike Becich (MB) and Bruce Friedman (BF)

Summary of Leica’s comments:1) DO - Leica participated in the European event and was please that they won awards for their

products at the event.2) DO and DG thought the European event was well communicated and went well but questioned

the value of some metrics.3) DO thought the document was clear and the scenarios and technical setup were useful.4) DO thought the document needed more details on the metrics and quantification of the

measures of the event.5) DO wanted clarification on the scenarios and the timing (wasn’t clear). MB explained the

scenarios and order of the vendors would be shuffled. BF very much liked MB’s idea of having the companies work different scenarios sequentially: vendor #1 works scenario #2; followed by vendor #2 works scenario #1, etc. No vendor then is in an unfavorable position in terms of the order of presentation. However, this causes problems if only one vendor is on the stage at any time with the rest in the bullpen. Lot's of confusing set-up time on the stage. Could get tedious. It seems to me that the only way this would work if all three (or four) vendors are continuously positioned on the stage at the same time with the "spotlight" on each as they go through each scenario. Then, only the switches need to be reset for each vendor.

6) DG what is the output of the showcase....need to finalize. MB stated that this would be proposed in the next draft of the document. Frozen section scenario would have 3 metrics -

7) DO queried on who would run the showcase and MB stated that a pathologist and a technical assistant who used the system would be running the company’s system and not vendor staff.

8) BF asked about the second event was canceled in Europe. DO did now know why Helsinki event was canceled.

9) DO thought the “end to end” scenarios and focus on the workflow was an appropriate one. The focus on how the digital pathology wholistically fits into their laboratory is a good and clean goal.

10) DO thought the technical setup was robust and adequate. Leica wanted gigabit between the image server and the viewing software. Include the make, model and specification of .

11) DO mentioned the potential of include a wireless broadcast mode during the showcase (very interesting). BF really liked this idea having the images available wirelessly for the audiences with iPads and smartphones.

12) DG asked for a full day instead of 8 to 10 hours to get the work pods for the showcase setup and testing. There will need to be able to do this in “private” to protect company intellectual property (IP). Sturdy tables should be available so that vendors can set up in a private area and then have their setup “wheeled” into place. BF thought that we should consider having set-up the previous afternoon/evening and the morning of the showcase. The showcase would then start at 1:00 on the workshop day and extend to perhaps 5:00 for a total of four hours (with perhaps a break). This would allow registrants for the event to fly in to Chicago that morning.

13) BF thought the event should be in the afternoon to allow for same day travel for participants and to give more time for setup. BO agreed that this would be ideal.

14) BO thought having a 5 to 10 minute time for the vendor to speak during the opening of the event perhaps in “dead-time” (during scanning perhaps). DG agreed but thought this would be

better at the end of the event. BF thought a closing remarks or panel Q&A would be useful at the end of the workshop. I think perhaps that we should have a wrap-up at the end with the thee reps at a table at the end to answer questions and wrap-up. Given the three hour time limit, I think that this may be difficult with four vendors.

15) BO thought the company’s selected pathologist would be a valuable spokesman and should be drawn into discussion and commentary.

16) BO thought we should be clear on what will be published and where it would be available and to whom.

17) DG thought approaching media and medical societies (many housed in Chicago) about the event would be useful.

18) DG wanted to make sure this was fair to all participants and having more details on the output of the event was critical.

19) BF thought we should attempt to videotape the entire session and make this available on the web site. Documenting the entire event would be of value to the participating companies.

20) BF commented on the potential name of the Vendor Showcase and taking into consideration the emphasis on end-to-end demonstrations and the Leica emphasis on workflow, I think that we should refer to this event going forward as: Vendor Showcase: End-to-End Digital Pathology Workflow.

Interviews with Industry Partners

John Fonte (JF), MetasystemsJuly 26th, 2011Interviewers – Mike Becich (MB) and Bruce Friedman (BF)

Summary of John Fonte’s comments:1) Overall JF thought the idea of a digital pathology showcase as proposed in current Draft is a

good idea, especially the “open” and “consensus building” for the showcase.2) JF thinks the focus on end to end solutions from scanning, to display and diagnose is a good one

for the industry.3) JF stated Metasystems has a slightly different take on systems looking at FISH, ISH, fluorescence

and molecular testing on whole slides and this can be done remotely by matching an identified area on light microscope images to areas for analysis at a remote site without pathologist having to re-identify areas in the special testing suite.

4) JF would like to see this (see #3 above) as a case scenario.5) JF despite the company’s focus as stated above, they think the showcase is an important event

and they (Metasystems) supported the European scanner event as a participant.6) JF thought that having pathologists and technologists running the demos was risky but that

generally valuable.7) JF was pleased with the technical setup and lead time we promised provide.8) JF was going to circulate the documents who were more involved in the European contest.

Interviews with Industry Partners

Rajiv Enand (RE), Omnyx, LLCAugust 13th, 2011Interviewers were Mike Becich (MB) and Bruce Friedman (BF)

1) RE sounds like a fantastic event and put together nice ideas which will help industry and pathologists alike.

2) Sept 2012 would be a great time for an event like this in terms of market acceptance3) Wanted to show commercial development if possible.4) RE asked about venue – Hyatt Grand in Chicago5) BF reiterated about having color commentary on the stage during the event.6) RE wanted to know if this was a “contest” or a demonstration. BF and MB reiterated that this

would be a “showcase” more like a demo but emphasizing end to end integrated workflow.7) MB told about fall backs – specs, demo set up, etc...8) RE felt that videotaping was of little commercial value to vendors. Marketing value is how many

participants in the venue.9) RE had questions on marketing the Industry Showcase – simulcast on web would be a good idea.

Vendors like Omnyx would help to promote the event.10) MB committed that there would be nothing running against this event.11) RE commented on the amount of time the companies would have to spend to prepare for this

so everything Path Info did to drive both attendance in person and on-line would be very important to get participation.

12) RE thought this is an important event to encourage adoption and provide this as a valuable service to the pathology community.

13) RE said this would be of great interest to Omnyx.14) RE thought a “judges choice” award would be a great idea.15) RE agreed that image quality would be a useful metric worthy of an award or recognition.16) RE thought that a narrative assessment by judges that were quotable document published as

proceedings in Journal of Pathology Informatics.17) RE thought we should try to get unbiased judges who would use the technology.