An evaluation of the Brazilian Mortality Information System · [email protected] 2...

ABSTRACT This paper assesses quality indicators of the Mortality Information System (SIM) with professionals and managers. Each indicator was analyzed with use of Stochastic Oscillator for qualitative evaluation of Likert scales. The evaluation indicated that the system has features that meet users, is effective to the management, provides adequate performance, enables auditing and tracking of accesses and operations, has mechanisms that ensure disas-ter recovery situations in data, and has a robust interface. Interoperability indicators were poorly evaluated, confirming reports on the lack of integration between the health informa-tion systems of the Unified Health System, fragmentation, and duplication of information.

KEYWORDS Health management. Mortality registries. Health evaluation.

RESUMO O artigo avalia indicadores de qualidade do Sistema de Informações sobre Mortalidade (SIM) com profissionais e gestores. Cada indicador foi analisado com uso de oscilador estocásti-co para avaliação qualitativa de escalas Likert. A avaliação indicou que o SIM possui funciona-lidades que atendem os usuários, é efetivo à gestão, apresenta desempenho adequado, possibilita auditoria e rastreamento de acessos e operações, possui mecanismos que garantem a recuperação de dados em situações de falhas e possui uma interface robusta. Os indicadores de interoperabili-dade foram mal avaliados, confirmando relatos sobre falta de integração, fragmentação e dupli-cidade de informações nos sistemas de informação de saúde do Sistema Único de Saúde.

PALAVRAS-CHAVE Gestão em saúde. Registros de mortalidade. Avaliação em saúde.

SAúDE DEBATE | rio de Janeiro, v. 41, n. especial, p. 101-117, Mar 2017

101

An evaluation of the Brazilian Mortality Information SystemUma avaliação do Sistema de Informações sobre Mortalidade

rinaldo Macedo de Morais1, andré lucirton costa2

1 Instituto Federal de São Paulo (IFSP), Departamento de Informática – Sertãozinho (SP), [email protected]

2 Universidade de São Paulo (USP), Faculdade de Administração, Contabilidade e Economia de Ribeirão Preto (Fearp) – Ribeirão Preto (SP), [email protected]

original article | artigo original

DOI: 10.1590/0103-11042017S09


Morais, r. M.; cosTa, a. l.102

Introduction

The gathering of mortality events in Brazil has improved significantly in recent decades; however, surveys on coverage and com-pleteness of vital events have classified the country in intermediate ranges (MaHaPatra et

al., 2007, MatHerS et al., 2005), with a significant volume of registries with undefined causes (liMa; QUeiroZ, 2011, FranÇa et al., 2013) and prob-lems in death notification and information flow (SiViero et al., 2013, FigUeiroa et al., 2013).

The Mortality Information System (SIM) supports the collection, storage and management process of death registries, in Brazil, which is mandatory in all cities. Mortality data are periodically sent to the State Secretaries of Health and then to the Ministry of Health. The system also has a module available on the web, which accesses the national database for investigation regis-trations and consultations, such as infant and neonatal deaths and deaths of pregnant and fertile women. The historical data stored by the system produce indicators that subsidize managers through managerial consultations through situational dashboard health con-sultation apps in consolidated databases.

Studies in literature show deficiencies in the quality of public healthcare infor-mation systems in Brazil, such as the lack of integration, fragmentation and duplic-ity of information (tHaineS et al., 2009, DaMÉ et

al., 2011), poor coverage of some systems and uncertainties regarding the reliability of the data they maintain (DaMÉ et al., 2011, BarBUScia;

roDrigUeS JÚnior, 2011, FariaS et al., 2011, Mota, 2009)

and deficiencies in support to the manager in decision-making and planning processes (Mota, 2009, ViDor; FiSHer; BorDin, 2011). Healthcare information in Brazil has multiple sources, poor quality of data and availability in formats that hinder their appropriation by managers and by social control (MoraeS, 2010) and monitoring of data quality in healthcare information systems that serve the Unified Healthcare System (SUS) do not follow a

regular evaluation plan (liMa et al., 2009).

This paper introduces an evaluation process for quality attributes for SIM with two user profiles: healthcare professionals who use the system to feed death registra-tions in cities in the Northeast of the state of São Paulo, and healthcare managers who analyze system information for management and decision-making in the same region.

Methods

Characterized as a cross-sectional study, this research consisted of a quality evaluation of the SIM that included healthcare profes-sionals from a macro-region of the state of São Paulo and health managers who work at the local, regional and state levels. The option to evaluate the SIM occurred because it is a high capillarity system, of regular and obligatory use in all Brazilian cities and that it has demanded many studies in public healthcare literature.

The evaluation with healthcare profes-sionals occurred through interviews in cities from the 13th Regional Healthcare Network of the State of São Paulo (RRAS-13), which includes the healthcare regions of Araraquara, Barretos, Franca and Ribeirão Preto. City selection criteria adopted a grid with five population ranges, adapted from the Brazilian Institute of Geography and Statistics (IBGE) municipal social indicator grid. Among the 90 cities in the macro region, 18 were selected, observing the proportional population distribution for the region by range, resulting in two cities with up to five thousand inhabitants (Cássia dos Coqueiros and Santa Cruz da Esperança), nine between 5 and 25 thousand (Dumont, Santo Antônio da Alegria, Serra Azul, Luiz Antônio, São Simão, Pradópolis, Miguelópolis, Cajuru and Santa Rosa de Viterbo), four between 25 and 50 thousand (São Joaquim da Barra, Ituverava, Barrinha and Santa Rita do Passa Quatro), two between 50 and 200 thousand


an evaluation of the Brazilian Mortality information System 103

(Taquaritinga and Barretos) and one above 200 thousand inhabitants (Ribeirão Preto). The cities in each range were selected out of convenience, including those in which pro-fessionals agreed to participate in surveys and who had been using SIM for at least three months.

The interviews with healthcare managers included five professionals who worked in the cities of Ribeirão Preto, Barretos, Araraquara, São Joaquim da Barra and Taquaritinga; four who worked in the Epidemiological Surveillance Groups (GVEs) in the Regional Healthcare Departments of Ribeirão Preto, Araraquara, Franca and Barretos; and a SIM manager from the São Paulo State Secretary

of Health Center for Strategic Information on Healthcare Surveillance Office. Selected professionals have worked with SIM for at least six months.

The model proposed by Morais and Costa (2014), designed as an instrument for evaluat-ing the quality of SUS information systems on a national scope, was used as a theoreti-cal reference for the evaluation process. This model includes three quality dimensions: software product quality, quality in use and service quality. For each dimension, the model specifies a set of quality indicators for health information systems. The structure and definition of the model’s quality attri-butes are described in chart 1.

Dimension Characteristics Sub-characteristics

product quality

Functional supportability: capacity of the software product to provide func-tions to meet explicit and implicit needs for which it has been conceived.

Functional completeness: capacity of the software product to provide an appro-priate set of functions for specified tasks and objectives of the user.

Functional correctness: capacity of the software product to provide, with the necessary degree of precision, correct or as agreed results or effects.

Functional adequacy: capacity of the software product to facilitate the performance of user tasks and objectives.

performance efficiency: capacity of the software product to maintain a proper level of performan-ce when used in specified conditions.

Behavior in relation to time: capacity of the software product to provide fitting response and processing times, when the software performs its functions under established conditions.

Use of resources: capacity of the software product to use fitting types and quantities of resources, when performing its functions under established conditions.

capacity: Maximum limits of system parameters (items that can be stored, number of competing users, bandwidth, transaction speed, size of database etc.) that meet requirements.

compatibility: capacity of the software product to enable an exchange of information with other applications and/or share the same hardware or software environment.

coexistence: capacity of the software product to coexist with other inde-pendent software products in a common environment and sharing common resources.

interoperability: capacity of the software product to interact with one or more specified systems through an exchange of information and the use of the information that is exchanged.

Chart 1. Structuring and definition of quality attributes of the evaluation model



product quality

Usability: capacity of the software product to be understood, learned, ope-rated and attractive to the user, effectively and effi-ciently, when used under specified conditions.

intelligibility: capacity of the software product to enable the user to unders-tand whether the software is fitting and how it can be used for specific-use tasks and conditions. depends on software documentation.

aprehendability: capacity of the software product to enable the user to learn how to use it. depends on software documentation.

operability: capacity of the software product to enable the user to operate and control it easily.

User error protection: capacity of the software product to protect the user from errors.

interface aesthetics with the user: capacity of the software product to be attractive to the user by offering an interface with pleasant interaction.

accessibility: capacity of the software product to be used by a broad spec-trum of people, including those with special needs and age-associated limitations.

reliability: capacity of the software product to execute its functions continuously.

Maturity: capacity of the software product to avoid failures resulting from defects in the software, maintaining normal operations.

availability: capacity of the software product to be operational and accessi-ble when its use is required.

Failure tolerance: capacity of the software product to operate at a specified level of performance in cases of defects in software or hardware.

recoverability: capacity of the software product to reestablish its specified level of performance and recover the data directly affected in the case of a failure.

safety: capacity of the software product to pro-tect information and data – unauthorized people or systems cannot read them or modify them and ac-cess to authorized people and systems is denied.

confidentiality: capacity of the software product to guarantee the data will be accessible only to people authorized to have access to them.

integrity: capacity of the software product to avoid unauthorized access for access to or modification of programs or data.

non-questioning: capacity of the software product to guarantee that the occurrence of actions or events can be proved, thus avoiding future ques-tioning.

responsibility: capacity of the software product to audit the traceability of access to operations.

authentication: capacity of the system to validate the identity of a user.

Maintainability: capacity of the software product to be modified. Modifi-cations can include cor-rections, improvements or adaptation of the software due to changes in the environment and in functional requirements or specifications.

Modularity: capacity of the system to have discrete components so a modi-fication in one component has minimal impact on other components.

reusability: capacity of software components to be used in other software or in the construction of other components/systems.

analyzability: capacity of the software product to allow the diagnosis of de-ficiencies or causes of failures, or the identification of parts to be modified.

Modifiability: capacity of the software product to allow implementation of a specified modification.

Testability: capacity of the software product to allow the validation of the software when it has been modified.

Chart 1. (cont.)



Two questionnaires oriented towards each user profile were prepared for collect-ing data, with evaluation questions for 45 quality indicators stipulated in the model specified by Morais and Costa. There were 28 questions for healthcare professionals who work in cities and 17 questions for man-agers, using four-point Likert scales.

The presentation of results used the ab-solute frequency distributions and percent-ages observed for the response categories for each indicator evaluated. The evaluation of results reused a procedure for data analysis with Likert scales, formulated by Sanches, Meireles and De Sordi (2011), with a version adapted from stochastic oscillator proposed

product quality

portability: capacity of the software product to be transferred from one environment to another.

adaptability: capacity of the software product to be adapted to different spe-cified environments, without the need for other actions or means besides those provided for this purpose by the software in question.

capacity to be installed: capacity of the software product to be installed in a specified environment.

capacity to replace: capacity of the software product to be used to replace another specified software product, with the same purpose and in the same environment.

Quality of use

effectiveness: capacity of the software product to allow users to achieve specified goals with accuracy and completeness in a specified use context.

efficiency: capacity of the software product to enable users to employ an appropriate quantity of resour-ces in relation to effectiveness obtained, in a specified use context.

satisfaction: capacity of the software product to satisfy users in a specified use context.

Utility: degree to which the user perceives the software helps in executing his activities.

credibility: degree of confidence the user has in the correct behavior of the system.

pleasantness: degree of user contentment in using the software.

comfort: degree of physical comfort in using the software.

absence of risk: capacity of the software product to present acceptable levels of risk of damage to peo-ple, business, property or the environment.

Mitigation of economic risk: degree in which the software reduces potential risk of a financial, operations, property or reputation nature in its context of use.

Mitigation of risks to safety and health: degree in which the software redu-ces potential risk in its context of use.

Mitigation of environmental risks: degree in which the software reduces potential risk to the environment in its context of use.

coverage of context: capacity of the software product to be used in its context of use and beyond those initially specified.

completeness of context: degree in which the software is used in all contexts of use (for example: using a low resolution monitor, with a low access to network rate, by an inexperienced user or without access to the network).

Flexibility: degree in which the software is used beyond its projected con-text of use (if software is not designed for flexibility, it may not be safe in unplanned contexts).

Quality of services: The degree to which the software product provides infrastructure for user support, in capacitation and support for the solution of problems regarding the system.

Chart 1. (cont.)

source: Morais e costa (2014).



by the authors, for evaluating the degree of agreement (DA) for each indicator, on a scale between zero and one, with the inclusion of

weight factors for the categories, as in the expression:

GC = { 100 – [ 100 ] } × 0.01

1 + (2 × ∑ CC + ∑ CP + 0.25 × ∑ DP + 0.00001)

(2 × ∑ DC + ∑ DP + 0.25 × ∑ CP + 0.00001)

In the expression, the sums for CA, PA, CD and PD refer to the totals obtained from compiling the answers for each category in the evaluation questionnaires (Completely Agree/ Partially Agree/ Completely Disagree/ Partially Disagree). The factor 0.00001 was added to the expression to

avoid errors in division by zero. As shown in table 1, the value calculated for the degree of agreement (DA) was mapped for agreement or disagreement very strong, substantial, moderate, low or negligible, adapted from the authors’ proposal for the adopted scale.

Table 1. Qualitative assessment of the degree of agreement (DA)

source: self elaboration.

DA value Qualitative interpretation

between 0.9 and 1.0 very strong agreement

between 0.8 and 0.9 substantial agreement

between 0.7 and 0.8 moderate agreement

between 0.6 and 0.7 low agreement

between 0.5 and 0.6 negligible agreement

between 0.4 and 0.5 negligible disagreement

between 0.3 and 0.4 low disagreement

between 0.2 and 0.3 moderate disagreement

between 0.1 and 0.2 substantial disagreement

between 0.0 and 0.1 very strong disagreement

An indicator was considered well evalu-ated for very strong, substantial or moderate agreement (DA value greater than 0.7) and poorly evaluated for very strong, substan-tial or moderate disagreement (DA lower

than 0.3). The result was considered incon-clusive for the other options (low or negli-gible agreements or disagreements, with DA values between 0.3 and 0.7).

This study was included in a project



submitted and approved by the Research Ethics Committee at the Hospital das Clínicas of the College of Medicine of Ribeirão Preto, University of São Paulo (CAAE/Conep:21337513.0.0000.5440) and there is no conflict of interest.

Results and Discussion

The evaluation with health professionals

Interviews with SIM users at the local level occurred between November 2013 and

February 2014 in the epidemiological sur-veillance departments of the secretaries of health in the selected cities. The application of the questionnaire was conducted person-ally by the researcher, with the presentation of each proposition associated with the indi-cator: the respondent indicated the options of answers, with opening for observations and comments.

Table 2 presents the frequency distributions for the answers to the questions that refer to the Software Product Quality (indicators 1 to 21), Quality in Use (indicators 22 to 25), and Quality of Services (indicators 26 to 28), with the respective evaluation, interpretation and result metrics for the evaluated indicators.

Indicator

Agree Desagree Metric(DA)

Qualitative interpretation

Evaluation of Indicatorcompletely partially partially completely

N % N % N % N %

1. The system has functions that provide and treat the information i need/ use.

13 72,2% 5 27,8% 0 0,0% 0 0,0% 0,96 very strong agreement

Well evalu-ated

2. The system presents correct and precise information.


Well evalu-ated

3. response time for an on-line operation in the system is satis-factory.

9 50,0% 8 44,4% 1 5,6% 0 0,0% 0,89 substantial agreement

Well evalu-ated

4. response time for a "job" request in the system is satis-factory.


Well evalu-ated

5. access and authentication in the system occur in satisfactory time.


Well evalu-ated

6. The system provides users documentation/ online assistan-ce for its functions.


Well evalu-ated

7. The learning of the system does not require long training.

3 16,7% 7 38,9% 3 16,7% 5 27,8% 0,48 negligible disagreement

inconclusive

8. The system's functions are easy to operate and intuitive.


Well evalu-ated

9. The system's functions have fast and standardized access/ navigation.


Well evalu-ated

Table 2. Distributions of frequency, metrics and evaluation of indicators – healthcare professionals



10. The system's functions pro-duce appropriate feedback, with clear messages.


Well evalu-ated

11. it is simple, easy and safe to correct an error in the system's functions: operations made by the user are reversible.


Well evalu-ated

12. The system's functions check whether data input values are valid.


Well evalu-ated

13. The system avoids the execu-tion of incorrect operations.

9 52,9% 5 29,4% 1 5,9% 2 11,8% 0,79 moderate agreement

Well evalu-ated

14. The user can adjust the layout of fields in the system function's interfaces to suit his or her work.

0 0,0% 0 0,0% 0 0,0% 15 100,0% 0,00 very strong disagreement

poorly evaluated

15. The system's functions have uniform and standardized inter-faces.


Well evalu-ated

16. The system includes features for users with special needs or the elderly.


poorly evaluated

17. The system undergoes much maintenance to correct errors.

3 18,8% 5 31,2% 5 31,2% 3 18,8% 0,50 negligible agreement

inconclusive

18. corrections, improvements or updates in the system cause ins-tability in the system or demand excessive effort or time.

4 25,0% 2 12,5% 4 25,0% 6 37,5% 0,40 negligible disagreement

inconclusive

19. errors occur during use of the system.

0 0,0% 5 29,4% 6 35,3% 6 35,3% 0,25 moderate disagreement

poorly evaluated

20. The system is available for the user, when requested.


Well evalu-ated

21. The system presents a low level of data loss and efficient mechanisms for restoration.


Well evalu-ated

22. The system offers adequate support/ features for executing my tasks/ activities.


Well evalu-ated

23. The system helps reduce my work time.

6 37,5% 7 43,8% 2 12,5% 1 6,2% 0,77 moderate agreement

Well evalu-ated

24. The use of the system faci-litates information storage and recovery.


Well evalu-ated

Table 2. (cont.)



FUNCTIONAL SUPPORTABILITY

The indicators for Functional completeness and Functional correctness, the first two in table 2, were very positively evaluated, without disagreement. Observe that for the Functional Correctness, 47.1% of users say SIM has some degree of incorrect informa-tion, especially related to causes of death. According to the respondents, this degree of incorrectness is primarily associated with registering the causes of mortality based on death certificates, which affect the quality of information maintained by the system.

With regard to the quality of data main-tained by the system, associated with func-tional correctness, records of deaths with undefined, erroneous or incomplete causes are frequently reported in the literature (FranÇa et al., 2013, BarBUScia; roDrigUeS JÚnior, 2011,

coSta; FriaS, 2011).Advances in the improvement of data

quality in SIM include greater commitment by the medical professional (Mello-Jorge;

laUrent; gotlieB, 2007, caScÃo; coSta; Kale, 2012) and by the professional who work at encrypt-ing and feeding the system (coSta; FriaS, 2011,

BraZ et al., 2013) and a better integration with other public agencies that have interface with healthcare information systems and services (Melo; ValongUeiro, 2015). One techni-cal improvement that can reduce incorrect inputs in the system is the inclusion of rules for validating motives of death related to gender, age and context of the death.

For premature deaths, Costa and Frias (2011) also report a strategy that has proven effi-cient in improving how to complete the DCs: the action by Infant Death Surveillance and the Infant Death Prevention Committees, organized by public authorities and civil society, which promote the systematic in-vestigation and discussion of these cases of death within the ambit of cities.

PERFORMANCE EFFICIENCY

The Performance Efficiency characteristic was evaluated satisfactorily by the users, for the local module as well as for the SIM web module, for the three behavior indica-tors in relation to when they were analyzed, for on-line, lot and authentication opera-tions (indicators 3, 4 and 5 in table 2). The

25. i do not lose time in solving technical problems associated with the system.

5 35,7% 5 35,7% 2 14,3% 2 14,3% 0,68 low agree-ment

inconclusive

26. Training and qualification are offered for using the system.

3 16,7% 7 38,9% 4 22,2% 4 22,2% 0,50 negligible agreement

inconclusive

27. There is a sized and qualified team for effective support in sys-tem use in difficult situations.


Well evalu-ated

28. system support service inclu-des an on-line (hot line) and/or help-desk service structure.

3 17,6% 8 47,1% 5 29,4% 1 5,9% 0,63 low agree-ment

inconclusive

Tabela 2. (cont.)




performance problems reported by users mainly relate to deficiencies in the internal network structure and access to the Internet in some cities, but they do not compromise system operability.

USABILITY

For Usability indicators (questions 6 to 16), responses suggest that:

(a) the system has user accessible informa-tion; however, you cannot infer whether the system is easy to learn, without long training, with the support of available doc-umentation. These results can be partially justified because, while some users were trained and had the support of profes-sionals in learning system use, such as in encrypting cases of death, others learned without any resources, with deficient or no support,

(b) the system has good operability, with easy and standardized operation, access and navigation, and its functions offer adequate operations feedback to the user, for making corrections easily and helping the user to prevent errors, vali-dating input values and blocking invalid operations and

(c) although users stated that the system has uniform and standardized interfaces, interface aesthetics were poorly evaluated in user flexibility for adjusting field layouts, as well as accessibility, because there were no features for users with special needs or the elderly.

RELIABILITY

In evaluating Reliability attributes (ques-tions 17 to 21), users diverge about the fre-quency of maintenance for error correction, and the stability of new versions, and they

gave a negative evaluation to the prevalence of errors during system operation. Some factors that may be associated with the poor evaluation of these indicators, based on reports, include training and time of system use, available infrastructure for system op-eration, data quality and availability of user support.

Two other reliability indicators were well evaluated: results show an appropriate degree of availability for the system for the local and the web module for accessing the national database, and the system has reli-able backup and restoration resources that guarantee support in cases of failure.

QUALITY IN USE

The results of SIM effectiveness in support-ing user activities (questions 22 to 25) also suggest that the system helps reduce the professional’s work time and is efficient in information storage and recovery.

QUALITY OF SERVICES

The indicators of Quality of Services (questions 26 to 28), to evaluate the infra-structure to support the user in training and support to solve problems, presented the following results: (a) the majority of users declared themselves satisfied with the training and qualification of the users of the system, although part of this major-ity evaluates that it is not adequate and (b) the results for the indicators associated with the infrastructure for user service suggest that the technical support team for the system is effective and qualified to help professionals with difficulties in using the system, although the on-line/help-desk service structure is not ideal: users reported they were rarely readily served, the professional was not always available for providing service and there were often delays in feedback for solving pending issues.



EVALUATION OF HEALTHCARE MANAGERS

Interviews with healthcare managers who use SIM occurred between January and March 2014. The interviews took place in the coordinations of epidemiological surveil-lance departments of the municipal health departments (municipal managers), epide-miological surveillance groups of regional health departments (regional managers) and the Directorate of the Strategic Information Center on Health Surveillance of the State

Secretariat of Health (state manager), after a previous scheduling.

The tabulation of quantitative and quali-tative results for the questionnaires is shown in table 3, which includes the frequency dis-tributions for the answers to the questions that refer to the Software Product Quality (indicators 1 to 10), Quality in Use (indica-tors 11 to 16) and Service Quality (indicator 17), with the respective evaluation, interpre-tation and result metrics for the evaluated indicators.

Indicator

Agree Desagree Metric(DA)

Qualitative interpretation

Evaluation of Indicatorcompletely partially partially completely

N % N % N % N %

1 - The system provides the func-tions i need to support decision--making.


Well evalu-ated

2 - The system provides func-tions that obey legal information standards (cid10, drG, data transmission etc.)


Well evalu-ated

3 - The system's functions provi-de clinical/ healthcare documen-tation in a correct and complete manner.


Well evalu-ated

4 - system features integrate different areas/ departments.

5 50,0% 2 20,0% 1 10,0% 2 20,0% 0,69 low agree-ment

inconclusive

5 - The system generates clini-cal/ healthcare documentation with a satisfactory response time.


Well evalu-ated

6 - The system provides its data to other systems that need to access it.

0 0,0% 1 12,5% 2 25,0% 5 62,5% 0,11 substantial disagreement

poorly evaluated

7 - The system integrates with other systems through the ex-change of information and the use of the information that is exchanged.

0 0,0% 2 20,0% 2 20,0% 6 60,0% 0,15 substantial disagreement

poorly evaluated

8 - For system functions that require or demand customization by the user, the system permits adaptations to meet local/ speci-fic needs.


poorly evaluated

Table 3. Distributions of frequency, metrics and evaluation of indicators – healthcare managers



FUNCTIONAL SUPPORTABILITY

Indicators for functional completeness (questions 1 to 3) were positively evaluated, with a predominance of answers of agree-ment. The respondents commented some observations:

(a) There is a demand for decision-making reports that show consolidated data. Many reports are difficult to extract and do not contain summarized data, which requires

manager rework in processing data on elec-tronic spreadsheets;

(b) With regard to legal standards and standardized tables, inconsistencies were reported associated with CID10, Brazilian Classification of Occupations (CBO) and street addresses;

(c) Reports and consultations to the system do not include medical chart data and require manual investigation in

9. The system’s functions present clear interfaces (screens/forms/ data input/ reports/graphs), with understandable terms and wi-thout ambiguities.


Well evalu-ated

10. The system’s functions re-gister access and operations information that can be audited/ tracked in the future.


Well evalu-ated

11. i can obtain the information i need when using the system.


Well evalu-ated

12. The system effectively contri-butes to the strategic objectives of the management of professio-nals (medical, nursing, adminis-trative teams etc.).


Well evalu-ated

13. The system has been contri-buting to improving the quality of the offer of healthcare services.


Well evalu-ated

14. The system has been contri-buting to improving the manage-ment of healthcare services.


Well evalu-ated

15. different groups of users are satisfied with the system.


Well evalu-ated

16. The system operates accor-ding to its design and specifica-tions.


Well evalu-ated

17. The processing of change re-quests in the system is adequate (time, modifications, opening to user participation).

0 0,0% 4 44,4% 3 33,3% 2 22,2% 0,37 low disagree-ment

inconclusive

Table 3. (cont.)




other systems.Indicator 4, for the Functional Adequacy

was inconclusive: although 50% of those in-terviewed completely agree SIM integrates the areas/departments encompassed by the system, 30% indicated a low degree of in-tegration such as between teams that work integrated with grievance and notification teams, that uses Sinan system.

PERFORMANCE EFFICIENCY

SIM users evaluated the Performance Efficiency characteristic satisfactorily, with 100% in agreement, as suggested by the results for indicator 5. Performance prob-lems pointed out by users basically refer to deficiencies in network structure and access to the Internet, but they do not compromise system operation.

COMPATIBILIDADE

Compatibility indicators – Coexistence and Interoperability – were poorly evaluated by respondents, with 87.5 and 80% in disagree-ment, respectively, according to results for indicators 6 and 7. The users reported that:

- There is no integration with applica-tions that register clinical data for the dead person (as a patient);

- There is not integration with the Information System for Notifiable Diseases (Sinan);

- There are deficiencies in integration with the Live Births System (Sinasc);

- There are demands for more effective in-tegration between the SIM local and web modules;

- Crossing data for managerial reports is obtained manually from several sources/ systems.

The respondents also observed the inte-gration between applications, through data-linkages, is impracticable due to restrictions in access to information laws, which limit the identification of public records on popu-lation data up to the district level. Despite this limitation, the development of some specific linkages could help manage access to integrated information from multiple sources.

The indicators for Compatibility attri-butes were poorly evaluated, which con-firms reports in literature about the lack of integration between applications, fragmen-tation and duplicity of information (tHaineS et

al., 2009, DaMÉ et al., 2011). Several actions by the Ministry of Health seek to promote regula-tions on Interoperability standards, such as Ordinance 2.073/2011, the institutionaliza-tion of CIINFO (Healthcare Information and Information Technology Committee), management and improvement of PNIIS (National Healthcare Information and Information Technology Policy) and stan-dardization of technologies (Ministry of Health Ordinances 2.466/2009, 2.072/2011 and 188/2012), but effective integration between applications is still a challenge to be solved.

USABILITY

Two Usability indicators (questions 8 and 9) were evaluated: (a) indicator 8, which checks system flexibility, was poorly evaluated by respondents, interviewed said the SIM does not have a modifiable interface, but rather a fixed standard based on forms approved by the system management committee; there-fore, there is no flexibility for customization; (b) indicator 9, associated with interface aesthetics, checks to what degree the system presents clear forms, reports, screens and graphics, with understandable and unam-biguous terms. It was evaluated positively, with 90% agreement.



SAFETY

The accountability and traceability of access to operations indicator (question 10) was evaluated positively: those interviewed said the system maintains logs of events executed by users at the local and the web module levels. Access controls are recorded with user data registered by managers who attend to safety requirements for the system.

QUALITY IN USE

For Quality in Use, effectiveness indica-tor – which refers to the degree the system permits users to access related information during use was positively evaluated, as well as the indicators for satisfaction, utility and coverage of context.

QUALITY OF SERVICES

A single indicator was evaluated for Quality of services (question 17): if the processing of modifications required for the system is ade-quate in relation to time, modification results and opening to user participation in progres-sive maintenance processes. In the results, 44.4% agree with the affirmation, whereas 55.6% disagree. The negative evaluations for this indicator were justified by respondents with arguments that the periods in which the required modifications are handled are very long; the modifications implemented do not always meet user needs and the decision on priorities does not involve users at the local and state levels.

Conclusions

Although the literature reports problems associated with coverage, incomplete data, integration and support for management, for SIM (FranÇa et al., 2013, SiViero et al., 2013, BarBUScia;

roDrigUeS JÚnior, 2011; Mota, 2009; BraZ et al., 2013), in absolute numbers, SIM was well evaluated

by both user profiles in this process. For the 45 attributes analyzed for system quality, there were 31 positive evaluations and six negative evaluations, while eight indicators presented divergent or inconclusive results.

The evaluation indicated that the SIM has features that meet user needs; it has ade-quate performance; and, despite deficiencies in adaptations and interface accessibility, usability was well evaluated. With regard to Reliability, the evaluation indicated the system is available and has mechanisms that guarantee data recovery in fault situa-tions; however, according to respondents, the system has a prevalence of errors and instability in new versions, which compro-mises normal operations. Safety was also well evaluated. The system enables auditing and tracking of accesses and operations per-formed by its users.

Interoperability indicators were poorly evaluated, confirming reports in the litera-ture about lack of integration, fragmentation and duplicity of information on informa-tion systems for SUS. The quality of data maintained by the system was also one of the problems pointed out by users, due to records of deaths with undefined, misin-formed or incomplete causes.

In relation to Quality in Use, the results of the analyzed indicators suggest the SIM has been effective in supporting management, with positive ratings above 70%.

For the Quality of Services dimension, the indicators revealed deficiencies in user support and service. The training and em-powerment of professionals who operate the SIM in cities was one of the demands identified in this study: the training of pro-fessionals was seen as heterogeneous and investments in qualification could bring gains in the quality of data maintained by the system.

User support infrastructure also lacks the organization of better-structured remote service that offers feedback that is more effective to the user. Results also indicate



there is no adequate management of modi-fications in the system, in relation to time, content of alterations and opening for user participation.

In short, it was possible to evaluate the SIM with its main users, identify the attri-butes the system has that are adequate and those that need improvement. The positive evaluation for SIM does not extend to others applications. It must be remembered that SIM has considerable stability and that it was the first public healthcare information system implemented in Brazil, back in 1976.

In a recent study, Santos and Teixeira (2016) carried out a systematic review of studies in the literature related to health

policies within the scope of the Ministry of Health between 1998 and 2014. Among the findings of the research, only 2 articles on in-formation systems were found in a universe of 377 studies (0.5%), which shows the lack of literature in this area and a potential for future research.

The process described in this paper may help other evaluation projects for health-care information systems. It may be totally or partially replicated for other applica-tions and be useful as another reference for studies that involve evaluation processes of the technical quality of software in health-care or other areas, with adaptations. s



References

BARBUSCIA, D. M.; RODRIGUES JÚNIOR, A. L.

Completude da informação nas declarações de nascido

vivo e nas declarações de óbito, neonatal precoce e

fetal, da região de Ribeirão Preto, São Paulo, Brasil,

2000-2007. Cad. Saúde Pública, Rio de Janeiro, v. 27, n.

6, p. 1192-1200, jun. 2011.

BRAZ, R. M. et al. Avaliação da completude da variável

raça/cor nos sistemas nacionais de informação em

saúde para aferição da equidade étnico-racial em

indicadores usados pelo Índice de Desempenho do

Sistema Único de Saúde. Saúde debate, Rio de Janeiro,

v. 37, n. 99, p. 554-562, 2013.

CASCÃO, A. M.; COSTA, A. J. L.; KALE, P. L. Qualidade

da informação sobre mortalidade numa coorte de

diabéticos - Estado do Rio de Janeiro, 2000 a 2003. Rev.

bras. epidemiol., São Paulo, v. 15, n. 1, p. 134-142, mar.

2012.

COSTA, J. M. B. S.; FRIAS, P. G. Avaliação da

completitude das variáveis da declaração de óbitos de

menores de um ano residentes em Pernambuco, 1997-

2005. Ciênci saúde coletiva, Rio de Janeiro, v. 16, supl. 1,

p. 1267-1274, 2011.

DAMÉ, P. K. V. et al. Sistema de Vigilância Alimentar

e Nutricional (SISVAN) em crianças do Rio Grande

do Sul, Brasil: cobertura, estado nutricional e

confiabilidade dos dados. Cad. Saúde Pública, Rio de

Janeiro, v. 27, n. 11, p. 2155-2165, nov. 2011.

FARIAS, L. M. M. et al. Os limites e possibilidades do

Sistema de Informação da Esquistossomose (SISPCE)

para a vigilância e ações de controle. Cad. Saúde

Pública, Rio de Janeiro, v. 27, n. 10, p. 2055-2062, out.

2011.

FIGUEIROA, B. Q. et al. Análise da cobertura do

Sistema de Informações sobre Mortalidade em Olinda,

Pernambuco, Brasil. Cad. Saúde Pública, Rio de Janeiro,

v. 29, n. 3, p. 475-484, mar. 2013.

FRANÇA, E. B. et al. Strengthening vital statistics in

Brazil: investigation of ill-defined causes of death

and implications on mortality statistics. The Lancet,

London, v. 381, supl. 2, p. 51-51, jun. 2013.

LIMA, C. R. A. et al. Revisão das dimensões de

qualidade dos dados e métodos aplicados na avaliação

dos sistemas de informação em saúde. Cad. Saúde

Pública, Rio de Janeiro, v. 25, n. 10, p. 2095-2109, out.

2009.

LIMA, E. E. C.; QUEIROZ, B. L. A evolução do sub-

registro de mortes e causas de óbitos mal definidas em

Minas Gerais: diferenciais regionais. Rev. bras. estud.

popul., Rio de Janeiro, v. 28, n. 2, p. 303-320, dez. 2011.

MAHAPATRA, P. et al. Civil registration systems and

vital statistics: successes and missed opportunities. The

Lancet, London, v. 370, n. 9599, p. 1653-1663, nov. 2007.

MATHERS, C. D. et al. Counting the dead and what

they died from: an assessment of the global status of

cause of death data. Bull World Health Organ, Genebra,

v. 83, n. 3, p. 171-177, mar. 2005.

MELLO-JORGE, M. H. P.; LAURENTI, R.; GOTLIEB,

S. L. D. Análise da qualidade das estatísticas vitais

brasileiras: a experiência de implantação do SIM e do

SINASC. Ciência saúde coletiva, Rio de Janeiro, v. 12, n.

3, p. 643-654, jun. 2007.

MELO, G. B.; VALONGUEIRO, S. Incompletude dos

registros de óbitos por causas externas no Sistema

de Informações sobre Mortalidade em Pernambuco,

Brasil, 2000-2002 e 2008-2010. Epidemiol. Serv. Saúde,

Brasília, DF, v. 24, n. 4, p. 651-660, dez. 2015.

MORAES, I. H. S. Sala de situação em saúde:

contribuição à ampliação da capacidade gestora do

Estado? In: ORGANIZAÇÃO PAN-AMERICANA DA

SAÚDE. Sala de situação em saúde: compartilhando as

experiências do Brasil. Brasília, DF: Organização Pan-

Americana da Saúde, Ministério da Saúde, 2010.

MORAIS, R. M.; COSTA, A. L. Um modelo para

avaliação dos sistemas de informação do SUS de

abrangência nacional: o processo de seleção e



estruturação de indicadores. Revista de Adm. Pública,

Rio de Janeiro, v. 48, n. 3, p. 377-385, jun. 2014.

MOTA, F. R. L. Registro de informação no sistema de

informação em saúde: um estudo das bases SINASC,

SIAB e SIM no estado de Alagoas. 2009. Tese

(Doutorado em Ciência da Informação) – Escola da

Ciência da Informação, Universidade Federal de Minas

Gerais, Belo Horizonte, 2009.

SANCHES, C.; MEIRELES, M.; DE SORDI, J. O.

Análise qualitativa por meio da Lógica Paraconsistente:

método de interpretação e síntese de informação obtida

por escalas Likert. In: ENCONTRO DE PESQUISA

EM ADMINISTRAÇÃO E CONTABILIDADE DA

ANPAD (ENEPQ), 3., 2011, João Pessoa. Anais... João

Pessoa, 2011. Disponível em: <http://www.anpad.org.

br/diversos/trabalhos/EnEPQ/enepq_2011/ENEPQ221.

pdf>. Acesso em: 24 abr. 2014.

SANTOS, J. S.; TEIXEIRA, C. F. Política de saúde no

Brasil: produção científica 1988-2014. Saúde debate, Rio

de Janeiro, v. 40, n. 108, p. 219-230, jan./mar. 2016.

SIVIERO, P. et al. Indicador de subnotificação de

óbitos no Sistema de Informação de Mortalidade no

Brasil obtido de pacientes que morreram por doença

renal crônica terminal: mensuração baseada nas

Autorizações de Procedimentos de Alta Complexidade

de 2000 a 2004. Cad. saúde colet., v. 21, n. 1, p. 92-95,

mar. 2013.

THAINES, G. H. L. S. et al. Produção, fluxo e análise

de dados do sistema de informação em saúde: um caso

exemplar. Texto & Contexto Enfermagem, Florianópolis,

v. 18, n. 3, p. 466-474, set. 2009.

VIDOR, A. C.; FISHER, P. D.; BORDIN, R. Utilização

dos sistemas de informação em saúde em municípios

gaúchos de pequeno porte. Revista de Saúde Pública,

São Paulo, v. 45, n. 1, p. 24-30, 2011.

received for publication: april 2016 Final version: october 2016 conflict of interests: non-existent Financial support: non-existent

Date post:	30-Nov-2018
Category:	Documents
Upload:	vothuy
View:	212 times
Download:	0 times

An evaluation of the Brazilian Mortality Information System · [email protected] 2...

Documents