1
PERSIVAL
a System for Personalized Search and Summarization over Multimedia
Information
PERSIVAL
a System for Personalized Search and Summarization over Multimedia
Information
2
PERSIVAL team membersPERSIVAL team members Medical InformaticsMedical Informatics
James Cimino, Carol Friedman, Steven JohnsonJames Cimino, Carol Friedman, Steven Johnson
Medical School – cardiac anesthesiologyMedical School – cardiac anesthesiology Desmond JordanDesmond Jordan
Computer ScienceComputer Science Steven Feiner, Luis Gravano, Vasileios Hatzivassiloglou, Kathleen McKeownSteven Feiner, Luis Gravano, Vasileios Hatzivassiloglou, Kathleen McKeown
Electrical EngineeringElectrical Engineering Shih-Fu ChangShih-Fu Chang
Center for Research on Information Access, Health Sciences Center for Research on Information Access, Health Sciences LibraryLibrary
Judith Klavans, Pat Molholt, Elizabeth LaRue, David MillmanJudith Klavans, Pat Molholt, Elizabeth LaRue, David Millman
Cognitive ScienceCognitive Science Andre Kushniruk (York), Vimla Patel (Medical Informatics)Andre Kushniruk (York), Vimla Patel (Medical Informatics)
3
StudentsStudents
Computer ScienceComputer Science Eugene AgichteinEugene Agichtein Michel GalleyMichel Galley Noemie ElhadadNoemie Elhadad Panos IpeirotisPanos Ipeirotis
Medical InformaticsMedical Informatics Michael Charney (programmer)Michael Charney (programmer) Eneida MendoncaEneida Mendonca Lyudmila Shagina (programmer)Lyudmila Shagina (programmer)
Electrical EngineeringElectrical Engineering Shahram EbadollahShahram Ebadollah
Min-Yen KanMin-Yen Kan Simon LokSimon Lok Smaranda MuresanSmaranda Muresan
Sergey SigelmanSergey Sigelman (programmer)(programmer)
Yoon –Ho SeolYoon –Ho Seol Di WangDi Wang
4
GoalsGoals
Personalized access to distributed, multimedia Personalized access to distributed, multimedia resourcesresources
information access information access information fusioninformation fusion information understandinginformation understanding
Provision of patient-specific informationProvision of patient-specific information interaction within contextinteraction within context for clinicians, at the point of patient carefor clinicians, at the point of patient care for patients, in terms that can be understoodfor patients, in terms that can be understood online patient record serves as a user modelonline patient record serves as a user model
5
RoundsRounds
Patient-centricPatient-centric Current: Access Current: Access
to clinical datato clinical data Missing: Missing:
Access to Access to literature that literature that fits patient fits patient profileprofile
6
Unique ContributionsUnique Contributions
System focus: querying, search, presentationSystem focus: querying, search, presentation Questions are asked within the context of patient Questions are asked within the context of patient
informationinformation A uniform, personalized view of distributed resources A uniform, personalized view of distributed resources
on the internet through querying and browsingon the internet through querying and browsing Concise, patient specific presentation of relevant Concise, patient specific presentation of relevant
information through summarizationinformation through summarization Access to textual documents linked with access to Access to textual documents linked with access to
multimedia video: library of echocardiogrammultimedia video: library of echocardiogram Dynamic layout of heterogeneous informationDynamic layout of heterogeneous information
7
Where are we now?Where are we now? Prototypes of each system componentPrototypes of each system component Local library of journal articles and consumer health Local library of journal articles and consumer health
sitessites 20 highly ranked journals20 highly ranked journals 30,000 articles30,000 articles
Facilities for distributed online searchFacilities for distributed online search Scenarios for development and testing with three Scenarios for development and testing with three
patientspatients Initial system integrationInitial system integration
Restricted to a limited set of examplesRestricted to a limited set of examples
Formative evaluation of system componentsFormative evaluation of system components
8
Overall Integrated DemoOverall Integrated Demo
What is the prognosis for atrial fibrillation and What is the prognosis for atrial fibrillation and myocardial infarction?myocardial infarction?
Clinician as user Clinician as user On viewing patient discharge summaryOn viewing patient discharge summary Journal articles: controlled clinical trialsJournal articles: controlled clinical trials Re-ranking of search results using patient recordRe-ranking of search results using patient record
What is the treatment for endocarditis? What is the treatment for endocarditis? Patient as user Patient as user On viewing lab resultsOn viewing lab results Consumer health informationConsumer health information
11
User Interface FocusUser Interface Focus
Asking questions within context of patient Asking questions within context of patient recordrecord
Evidence based medicine to suggest Evidence based medicine to suggest questionsquestions
Selection of relevant information from the Selection of relevant information from the patient recordpatient record
Demo of MedleeDemo of Medlee
13
Distributed SearchDistributed Search
Meta-searcher for automated interaction with Meta-searcher for automated interaction with heterogeneous, distributed sourcesheterogeneous, distributed sources
Use of machine learning and query probes to Use of machine learning and query probes to automatically determine topics of distributed automatically determine topics of distributed sourcessources
Information extraction from web pagesInformation extraction from web pages
15
Re-ranking search resultsRe-ranking search results
Re-rank articles which better match the Re-rank articles which better match the patient record -> more relevant articles patient record -> more relevant articles
Use natural language techniques to analyze Use natural language techniques to analyze article and patient recordsarticle and patient records
Articles with many terms and values Articles with many terms and values matching the patient record score highermatching the patient record score higher
17
Presentation FocusPresentation Focus
Multimedia summarizationMultimedia summarization Journal articles, consumer health, videoJournal articles, consumer health, video Highlight retrieved results to help user in finding relevant Highlight retrieved results to help user in finding relevant
informationinformation Personalize summary for patientPersonalize summary for patient Define unknown terminology Define unknown terminology Methods for summarizing and search echocardiogramsMethods for summarizing and search echocardiograms
Dynamic layout and organization of resultsDynamic layout and organization of results Explicitly control level of detailExplicitly control level of detail
18
MilestonesMilestones
Where we said we would be vs. where we are:Where we said we would be vs. where we are: Year 2: Year 2: skeletal end-to-end system prototypeskeletal end-to-end system prototype with with minimal minimal
personalization, interactivitypersonalization, interactivity, and , and limited coverage of structured limited coverage of structured documentsdocuments
Year 3: Year 3: Extend to full prototypeExtend to full prototype, with , with increased personalizationincreased personalization, , interactivityinteractivity, , limited coordination of multimedialimited coordination of multimedia, , full range of full range of structured documentsstructured documents, and , and restricted coverage of consumer documentsrestricted coverage of consumer documents
Use of evidence-based medicine, machine learning to categorize Use of evidence-based medicine, machine learning to categorize sources by topic, provision of definitions, thin-client computing to sources by topic, provision of definitions, thin-client computing to allow PERSIVAL on mobile, hand-held devicesallow PERSIVAL on mobile, hand-held devices
Year 4: Scale prototype with increased robustness, personalization, Year 4: Scale prototype with increased robustness, personalization, coverage to full range of documents and fully integrated multimedia. coverage to full range of documents and fully integrated multimedia. Coordinate with end-to-end evaluationCoordinate with end-to-end evaluation
Year 5: Refine components based on Year 4 evaluation. Transition Year 5: Refine components based on Year 4 evaluation. Transition PERSIVAL to deployment in cooperation with Health Sciences PERSIVAL to deployment in cooperation with Health Sciences LibraryLibrary
19
Plans for next yearPlans for next year
Increase robustnessIncrease robustness Extend question asking to different patient contexts, different Extend question asking to different patient contexts, different
question typesquestion types Allow summarization and re-ranking of online articlesAllow summarization and re-ranking of online articles Extend journal summarization to new genresExtend journal summarization to new genres Extend layout to dynamically incorporate different types of Extend layout to dynamically incorporate different types of
summary inputsummary input
Multimedia integrationMultimedia integration Implement scenarios for integration Implement scenarios for integration Increase interaction with video summary in layout Increase interaction with video summary in layout Enhanced multimedia prototypeEnhanced multimedia prototype