A Preliminary Systematic Literature Review of the use of Formal Methods
in Medical Software Systems
Silvia Bonfanti1,2, Angelo Gargantini1 and Atif Mashkoor2
1Università of Bergamo - Italy, 2Software Competence Center Hagenberg, Austria
Outline
• SLR (Systematic Literature Review) goals
• SLR: definition and features
• SLR Process we adopted
• Analysis and results
• Limitations and future work
• Conclusions
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Goals
• Provide guidance to researchers starting to work on this topic
• Assess the state of the art for researchers already working on this subject
• Suggest areas for further investigation
• Identify groups that still working in this field
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Systematic Literature Review - SLR
• SLR is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest.
• "Indeed, one of my major complaints about the computer field is that whereas Newton could say, "If I have seen a little farther than others, it is because I have stood on the shoulders of giants," I am forced to say, "Today we stand on each other's feet." Perhaps the central problem we face in all of computer science is how we are to get to the situation where we build on top of the work of others rather than redoing so much of it in a trivially different way. Science is supposed to be cumulative, not almost endless duplication of the same kind of things".
Richard Hamming 1968 Turning Award Lecture
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Features of SLR
• Define a review protocol
• Define search strategy to include as much of the relevant literature
• Document the SLR process clearly
• Define criteria to include and exclude papers
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Where has been applied SLR in Computer Science?
• Web Application testing• Serdar Doğan, Aysu Betin-Can, Vahid Garousi, Web application testing: A systematic literature review,
Journal of Systems and Software, Volume 91, May 2014,
• Software Engineering• Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, Stephen Linkman,
Systematic literature reviews in software engineering – A systematic literature review, Information and Software Technology, Volume 51, Issue 1, January 2009,
• Mutation Testing• Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE
Trans. Softw. Eng. 37, 5 (September 2011)
• Software product line• Vander Alves, Nan Niu, Carina Alves, George Valença, Requirements engineering for software product
lines: A systematic literature review, Information and Software Technology, Volume 52, Issue 8, August 2010,
• Many other fields …
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
SLR Process
Query: • TITLE(medical) AND (TITLE(software) OR TITLE(device*))
AND (TITLE(validation) OR TITLE(verification) OR TITLE(certification))
• KEY(medical) AND (KEY(software) OR KEY(device*)) AND (KEY(validation) OR KEY(verification) OR KEY(certification)) AND KEY(formal)
Research Questions (RQ)
merged the authors written in a different way
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
RQ1: Which is the trend of publications?
Probably the updating of publications is not finished yet
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
RQ2: Are there more publications in Journals or Conferences?
The number of journal in medical field is biggerthan the number of conference
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
RQ3: Which are the most important journals/conferences?
Name JOUR/CONF SCImago Rank # citations # publications
Artificial Intelligence in Medicine JOUR Q1 255 3
International Journal of Medical Informatics JOUR Q1 136 2
Lecture Notes in Computer Science CONF N/A 95 35
Proceedings of the IEEE JOUR Q1 36 1
IEEE Transactions on Software Engineering JOUR Q1 36 1
Journal of Biomedical Informatics JOUR Q1 33 3
Proceedings of the 8th ACM International Conference on Embedded Software, EMSOFT'08
CONF N/A 26 1
Computer JOUR Q1 26 1
Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems
CONF N/A 26 2
Annals of Internal Medicine JOUR Q1 22 1
IEEE Robotics and Automation Magazine JOUR Q1 21 1
Biomedical Optics Express JOUR Q1 20 1
Journal of Pharmaceutical and Biomedical Analysis JOUR Q1 20 1
Joint Workshop on High Confidence Medical Devices, Software, and Systems and Medical Device Plug-and-Play Interoperability, HCMDSS/MDPnP 2007
CONF N/A 20 2
Dec
reas
ing
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
RQ4: How many papers about this topic have been written by the same author?
Author # publications
Jones, P.L. 9Curzon, P. 8
Mangharam, R. 8Jiang, Z. 7
Masci, P. 7Thimbleby, H. 7
Pajic, M. 6
- Occasional contributors- New topic in the scientific community
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
RQ5: Which are the most cited publications?
!!! Scopus cannot identify self-citations
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
N° Publications # citations
1Fox, J., Johns, N., & Rahmanzadeh, A. (1998). Disseminating medical knowledge: the PROforma approach. Artificial intelligence in medicine, 14(1), 157-182.
182
2Lopez, D. M., & Blobel, B. G. (2009). A development framework for semantically interoperable health information systems. International journal of medical informatics, 78(2), 83-103.
83
3Ten Teije, A., Marcos, M., Balser, M., van Croonenborg, J., Duelli, C., van Harmelen, F., ... & Seyfang, A. (2006). Improving medical protocols by formal methods. Artificial intelligence in medicine, 36(3), 193-209.
56
4Maldonado, J. A., Moner, D., Boscá, D., Fernández-Breis, J. T., Angulo, C., & Robles, M. (2009). LinkEHR-Ed: A multi-reference model archetype editor based on formal semantics. International journal of medical informatics, 78(8), 559-570.
53
5Jiang, Z., Pajic, M., & Mangharam, R. (2012). Cyber–physical modeling of implantable cardiac medical devices. Proceedings of the IEEE, 100(1), 122-137.
36
6Cordeiro, L., Fischer, B., & Marques-Silva, J. (2012). SMT-based bounded model checking for embedded ANSI-C software. Software Engineering, IEEE Transactions on, 38(4), 957-974.
34
7Zhu, X., Fan, J. W., Baorto, D. M., Weng, C., & Cimino, J. J. (2009). A review of auditing methods applied to the content of controlled biomedical terminologies. Journal of biomedical informatics, 42(3), 413-425.
28
8Jetley, R., Iyer, S. P., & Jones, P. L. (2006). A formal methods approach to medical device review. IEEE Computer, 39(4), 61-67.
26
9Tschirner, S., Xuedong, L., & Yi, W. (2008, October). Model-based validation of QoS properties of biomedical sensor networks. In Proceedings of the 8th ACM international conference on Embedded software (pp. 69-78). ACM.
26
Some of papers fit not completely with the goal of our SLR
Depend on the keywords add by the authors and those added by Scopus
Limitation: why only Scopus?
Pro
• Scopus is the largest database owned by Elsevier, it contains scientific journals, books and conference proceedings.
• Scopus allows the user to perform different type of search, by title, by keywords, by authors or advanced search obtained by queries.
• All data have the same format
Con
• We excluded journals, conference papers, books,… that are not included in Scopus database.
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
… in the future
• Use other repositories e.g. Google Scholar, Web of Science
Problems:
- Some repositories do not provide an advanced search, or if it exists, it isnot always fully functional (e.g. IEEE, dblp)
- Some repositories contain large quantity of documents and it is difficult to select those important (e.g. Google Scholar includes technical reports, patents that are out of our scope)
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Limitation: keywords
Limitation
• Titles and keywords depends on the
authors choice:
Some papers were out of scope
We may have missed some important papers
Solution
• Extend the research to other sources
and check papers cited in selected
papers, in case they are not included
find a query to include them
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Limitation: low human interaction
Limitation
• Low degree of human interaction: we
used fields in the bibliographic entries
(year, type of publication ,…)
• In the statistics we have included
some papers that are out of scope
Solution
• Extend RQ with new that require a
deeper analysis of paper content:
• Which are the notations used?
• Which are the tools used?
• Which are the methodologies applied?
• Which are the typical case studies?
• Remove these papers manually after
reading them
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Limitation: metrics
Limitation
• Limited metrics to measure the impact
of the paper: number of citations
Solution
• Introduce new metrics like the
measure for citations using individual
h-index, which normalizes the number
of citations for each paper by dividing
the number of citations by the
number of authors for that paper, and
then calculate the h-index of the
normalized citation counts.
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Limitation: content of the papers
Limitation
• Some papers are an extended version
of conference, these should not be
included in the number of
contribution of the authors
• Some papers have the same content
(Pre-proceedings, proceedings)
Solution
• Remove manually these papers?
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Tools
In the prelimary analysis
• SciMAT:
A module dedicated to the management of the knowledge base and its entities
• RIS file
In the future
• SciMAT
• JabRef is an open source bibliography reference manager that allows also to group papers
• Script in Java to extract information from .bib files (e.g. papers that belong to a group defined in JabRef)
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Conclusions
• SLR about formal methods applied to medical device
• Run queries on Scopus
• We have answered to the following RQ:
• RQ1: Which is the trend of publications?
• RQ2: Are there more publications in Journals or Conferences?
• RQ3: Which are the most important journals/conferences?
• RQ4: How many papers about this topic have been written by the same author?
• RQ5: Which are the most cited publications?
• We defined future steps to improve the analysis
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
Silvia Bonfanti - A Preliminary Systematic Literature Review of the use of Formal Methods in Medical Software Systems
THANK YOU!!!QUESTIONS?
