Journal of Biomedical Informatics 51 (2014) 35–40

Contents lists available at ScienceDirect

Journal of Biomedical Informatics

journal homepage: www.elsevier .com/locate /y jb in

Predicting patient acuity from electronic patient records

http://dx.doi.org/10.1016/j.jbi.2014.04.0011532-0464/� 2014 Elsevier Inc. All rights reserved.

⇑ Corresponding author at: University of Turku, Department of Nursing Science,FI-20014 Turku, Finland. Fax: +358 10 5536179.

E-mail address: elina.kontio@turkuamk.fi (E. Kontio).

Elina Kontio a,b,⇑, Antti Airola c, Tapio Pahikkala c, Heljä Lundgren-Laine a,d, Kristiina Junttila e,Heikki Korvenranta d, Tapio Salakoski c, Sanna Salanterä a,d

a University of Turku, Department of Nursing Science, Finlandb Turku University of Applied Sciences, Finlandc University of Turku, Department of Information Technology, Finlandd Turku University Hospital, Finlande Hospital District of Helsinki and Uusimaa, Finland

a r t i c l e i n f o

Article history:Received 6 October 2013Accepted 1 April 2014Available online 12 April 2014

Keywords:Patient acuityPatient classification systemElectronic patient recordMachine learning

a b s t r a c t

Background: The ability to predict acuity (patients’ care needs), would provide a powerful tool for healthcare managers to allocate resources. Such estimations and predictions for the care process can be pro-duced from the vast amounts of healthcare data using information technology and computational intel-ligence techniques. Tactical decision-making and resource allocation may also be supported withdifferent mathematical optimization models.Methods: This study was conducted with a data set comprising electronic nursing narratives and theassociated Oulu Patient Classification (OPCq) acuity. A mathematical model for the automated assign-ment of patient acuity scores was utilized and evaluated with the pre-processed data from 23,528 elec-tronic patient records. The methods to predict patient’s acuity were based on linguistic pre-processing,vector-space text modeling, and regularized least-squares regression.Results: The experimental results show that it is possible to obtain accurate predictions about patientacuity scores for the coming day based on the assigned scores and nursing notes from the previousday. Making same-day predictions leads to even better results, as access to the nursing notes for the sameday boosts the predictive performance. Furthermore, textual nursing notes allow for more accurate pre-dictions than previous acuity scores. The best results are achieved by combining both of these informa-tion sources. The developed model achieves a concordance index of 0.821 when predicting the patientacuity scores for the following day, given the scores and text recorded on the previous day.Conclusions: By applying language technology to electronic patient documents it is possible to accuratelypredict the value of the acuity scores of the coming day based on the previous days assigned scores andnursing notes.

� 2014 Elsevier Inc. All rights reserved.

1. Introduction

One key question in a care process continuum is how to predictthe next steps within it. Crucially, being able to predict the careneeds of patients would help to reduce the rising costs of healthcare.The ability to predict acuity (patients’ care needs), would provide apowerful tool for health-care managers to allocate resources [1,2].

Tactical decision-making and resource planning in hospitalsincludes short- and medium-range plans, schedules andbudgets. Tactical decision-making also includes monitoring the

performance of organizational subunits, including departments,divisions, process teams, project teams and other workgroups[3,4]. Tactical resource planning in hospitals focuses on electivepatient admission planning and the intermediate-term allocationof resource capacities [5,6]. The main objectives of this planningare equitable access for patients, meeting production targetsand/or serving the strategically agreed number of patients, andefficiently using resources [7]. For example, in a perioperative unitthis might include the decisions to hire more staff to extend hours,expand the operating room capacity, purchase equipment, increaseblock time for a surgical group or to build a free-standing facility[8]. The objectives of capacity allocation are to balance surgicaland postsurgical resources [9–11] to maximize the contributionmargin per hour of surgical time [12].

36 E. Kontio et al. / Journal of Biomedical Informatics 51 (2014) 35–40

2. Background

2.1. Patient classification systems

One tool that can provide detailed clinical data for forecastingand real-time human resource allocation is the patient classifica-tion system (PCS). Patient classification can be determined bymethods and processes that are used to identify, validate andmonitor the needs of an individual patient [13–15]. Furthermore,the PCS provides information for human resource administration,accounting, budgeting and other functions of management[13,16].

A PCS assesses and classifies patients according to their acuity,their need of care, as well as the nursing activities that are neces-sary to fulfill those care needs during a certain time period [13].PCSs play an important role in supporting nurse managers’ deci-sion-making in organizing the care process and required resources.They also provide information on resource consumption, and assistin the budget planning for nursing services and care quality evalu-ation [13,16]. In addition, they are used to optimize availableresources and provide estimations of nurse-to-patient ratios[13,17].

1 Approval of the ethical committee of the hospital district (number 12/2009). Theethical discussion in this study is centered on the process of obtaining the necessarypermissions to carry out the research and to use the electronic documents. Inaddition, this study followed the security procedures designed for accessing patientdata [34].

2.2. Oulu Patient Classification system

Countless PCSs are currently in use globally. In Finland, themost widely used classification in inpatient units is the OuluPatient Classification (OPCq), which was developed on the basisof the HSSG Hospital Systems Study Group classification (HSSG)from 1991 to 1993 in Oulu University Hospital. The reliabilityand validity of the OPCq has been previously tested [18–20]. Usingthe OPCq and a nurse resource registry, it is possible to calculatethe nursing intensity per nurse: the nursing intensity indicatesthe nursing workload caused by the patients’ care needs (acuity).The OPCq is based on the principles of nursing presented in thequality control program and on Roper’s model of nursing [21].The OPCq scoring does not describe what specific tasks nurses havecompleted during the day, but how they responded to the patients’care needs with different processes and nursing interventions[22–24].

The OPCq [21,24] consists of six nursing care subsections: (1)planning and co-ordination of care; (2) breathing, blood circulationand symptoms of disease; (3) nutrition and medication; (4) per-sonal hygiene and excretion; (5) activity, movement, sleep andrest; (6) teaching, guidance during care and follow-up care, andemotional support. Each subsection is graded by the nurse dailyon a scale from A to D according to the patient’s care needs:A = 1 point, B = 2 points, C = 3 points and D = 4 points, resultingin a possible range of summarized OPCq scores from 6 to 24 points.The higher the score, the more demanding the need for care [23].Based on their OPCq score, patients are classified into five differentacuity categories: category I (6–8 points), category II (9–12 points),category III (13–15 points), category IV (16–20 points) and cate-gory V (21–24 points) [18,25,26]. A classification manual is avail-able to support this scoring process.

Data concerning patient acuity in Finland is one of the corenursing data belonging to the electronic patient record (EPR)[27]. According to the definition by the Organization for Standard-ization (ISO), the EPR is a repository of patient data in digital form,stored and exchanged securely that is accessible for multipleauthorized users. It contains retrospective, concurrent, and pro-spective information and its primary purpose is to support contin-uing, efficient and quality integrated health care. [28] However, asa method of data storage the EPR is complex, longitudinal andchallenging.

3. Contributions

The aim of this research is to study to what degree the clinicalinformation in the EPRs of cardiac patients can be used to predicttheir OPCq acuity scores for the following day. Our hypothesis isthat textual nursing notes and previously assigned acuity scorescan be utilized to predict the different sub-categories of a patient’sacuity for the next day through the application of machine-learning techniques. When evaluating our models, we considertwo distinct settings: one with the aim of accurately predictingacuity scores, and another where patients can be ranked fromthose needing the most care to those needing the least.

In recent years there has been significant interest in developingand applying text mining techniques based on machine learning tothe analysis of EPRs, leading to applications such as automateddiagnostic systems [29–31], text segmentation tools for nursingnarratives [32,33], and quality-of-life-prediction for patients [32].For a more thorough overview of research on text mining EPRs,we refer to [33]. To the best of our knowledge, the present studyis the first to address the problem of predicting patient acuityscores.

4. Methods

4.1. Data

The original data consists of 23,528 electronic patient records ofpatients with any type of heart problem that were admitted to auniversity hospital between 2005 and 2009. The data is collectedfrom six different information systems: text data from electronicpatient records; patients’ administrative data such as admission,discharge, transfer, patient acuity scores; text data from theradiology system and text data from the pathology system.

The inclusion criteria were:

– Diagnosis: ICD 10: I20-I25, I27, I30-I52, R00, R01.– Date of birth: 09/23/1901 – 10/21/2009.– Admitted into hospital between 2005 and 2009.– Length of hospital stay >1 day.

The data were further processed in order to connect the dailyacuity scores and the corresponding nursing documentation, sincethese data came from different information systems. For each twoconsecutive days of a hospital stay for the patient, a data point con-sisting of the text for the previous and following days, and theassociated acuity scores, was formed. After this pre-processing,the dataset consisted of 132,053 data points. The research wasconducted according to established ethical guidelines.1

4.2. Data representation

The textual records were lowercased, tokenized, and the wordswere reduced to their base forms using the FinTWOL morphologi-cal analyzer. A number of common Finnish stop words wereremoved from the texts; after this, the 10,000 most commonlyoccurring tokens were retained and the remaining tokens filteredout. Finally, the textual records were mapped into numerical vec-tors using the vector space model, defined as follows: Let us definethe term frequency of term t in textual record d, denoted as tft,d, asthe number of occurrences of the term in the record. Further, let us

2 Available at http://staff.cs.utu.fi/~aatapa/software/RLScore/.

E. Kontio et al. / Journal of Biomedical Informatics 51 (2014) 35–40 37

define the document frequency dft for a given term t as the numberof records in our collection containing the term, and the corre-sponding inverse document frequency as idft ¼ log N

dft, where N

denotes the overall number of records. Then, a term t in record dis assigned the weight as tf � idft,d = tft,d � idft. This approachassigns the more weight to a word the more frequently it appearsin a record, while at the same time assigning reduced importanceto very common words that are shared by a large number ofrecords (see Manning et al. [35] for a detailed description of vectorspace model and tf-idf weighting). Finally, each vector is normal-ized to Euclidean unit length.

By combining the vector space representation of the text andprevious acuity scores, we recovered the following feature setsconsidered in the experiments:

� ‘‘PScore’’, the acuity scores assigned to the patient on the previ-ous day.� ‘‘PText’’, the free text nursing notes written about the patient on

the previous day.� ‘‘Text’’, the textual nursing notes written about the patient on

the same day as the acuity scores are to be assigned.

The experiments where ‘‘PText’’ and/or ‘‘PScore’’ were used formaking predictions correspond to the setting where the aim wasto predict the acuity scores for the following day. In the settingwhere the ‘‘Text’’ features were also used, we were no longer pre-dicting future scores, but instead simply assuming that the modelmight be used to automatically input acuity scores to free nursesfrom this task.

4.3. Predictive model

Next, we describe a method known in the machine learning andstatistics literature as the regularized least-squares (RLS) or theRidge regression method [36]. This method is one of the mostwidely applied algorithms in the area of machine learning andhas been applied in previous research, for example on the auto-mated analysis of electronic patient records or the automated clas-sification of intensive care nursing narratives [30].

Let d be the dimension of the vector space into which eachrecord is transformed, and let m denote the number of recordsabout which the scores are known at the time of the model con-struction. We assume that the score yi of the ith record obeys thefollowing linear relationship with the entries of the correspondingvector entries:

yi ¼Xd

j¼1

wjxi;j þ ei

where xi,j denotes the jth vector entry of the ith record, wj are the dunknown real-valued parameters of the model, and ei are noiseterms that do not depend linearly on the ith record. Accordingly,we formulate the problem of inferring the model parameters fromthe set of labeled records as finding the minimizer of the followingpenalized regression problem:

Xm

i¼1

yi �Xd

j¼1

wjxi;j

!2

þ kXd

j¼1

w2j

Here, the first term measures the regression error on the trainingdata, the second term is the so-called Ridge penalty, and k > 0 is ahyper-parameter controlling the trade-off between the two terms.The role of the second term is to penalize the model complexityto avoid overfitting the model to the training data. The minimizerof the objective function can be found by solving a linear systemof equations. In our experiments, we use an implementation of

RLS from the RLScore software package,2 which solves the linearsystem using conjugate gradient optimization, as described by Rifkinet al. [37].

4.4. Evaluation measures

Recalling the one-to-five scale of the acuity scores and one-to-four scale of the subcategory scores, given a real valued predictionfor a record made by the model, we round the prediction to its clos-est score value among the possible scores. Consequently, the pre-diction error is measured as the absolute difference between therounded prediction and the true score, indicating that for a recordwith a true score of 1, it is more wrong to predict 3 than to predict2. This reflects the need for predicting the absolute number ofresources required.

Given that the amount of certain available resources is fixed,and thus needs to be allocated among patients, relative acuity isa more interesting quantity than absolute acuity. Therefore, wealso measure the model’s ranking performance. In the simplestcase of two patients under consideration, the question would bewhich of them is likely to require more resources.

For this purpose, we use the concordance index (CI)[38] [39].The concordance index over a set of data is the probability thatthe predictions for two records with different scores randomlydrawn from the set are in a correct order. That is, the predictionfor the record with the larger score is larger than that for the recordwith the smaller score:

CI ¼ 1Z

Xyi>yj

dðfi � fjÞ;

where fi and fj denote the predictions made for the ith and jthrecord, and Z is a normalization constant equal to the number ofrecord pairs with different scores, and d is the Heaviside step func-tion, with d(z) = 1, d(z) = 0.5, and d(z) = 0, for z > 0, z = 0, and, z < 0,respectively. The CI for a random model is 0.5, and for a perfect pre-dictor 1.0.

CI can be interpreted as an extension of the area under the ROCcurve (AUC) [40] to tasks that have multiple ordered categories,and in cases where there are only two categories, they are equiva-lent. CI corresponds to an estimate of the probability that giventwo randomly chosen patients with different acuity scores, themodel is able to predict which one should be assigned the higherscore.

The distribution of the OPCq acuity scores in the data is visual-ized in Fig. 1. It shows that most patients are assigned to categories3 and 4 for the overall score. For the 6 subcategories rated on ascale of A–D, scores B and C are the most prevalent.

4.5. Evaluation

The same experimental procedure is performed separately foreach of the considered feature sets, and for each acuity category.We considered both the regression setting, where the aim is to pre-dict the scores exactly, and the ranking setting, where our aim is toorder the patients from those needing the most care to those need-ing the least.

In addition to the RLS models, we also present results for twosimple baseline approaches. The previous day baseline alwaysassigns the same acuity score for each category that was assignedto that category on the previous day. The majority voter baselinealways predicts, for each category, the class that appears mostcommonly in the training data.

Fig. 1. Histogram of the OPCq acuity scores in the data set.

38 E. Kontio et al. / Journal of Biomedical Informatics 51 (2014) 35–40

The experiments were performed using 5-fold cross-validation.The division was made at the patient level, meaning that there isno overlap between the patients in the different folds. The regular-ization parameter was chosen from an exponential grid. On eachround of cross-validation, models corresponding to differentchoices of the regularization parameter were trained on threetraining folds, and the parameter chosen was the one that providedbest performance on the fourth training fold. Finally, the modelwas re-trained on the four training folds, and predictions madeon the test fold. The final performance was computed by compar-ing the predicted and true intensity scores on the test folds. For thecross-validated results, 95% confidence intervals were computed.Since the concordance index is computed over pairs of predictions,we randomly chose a subsample of pairs from the test folds whereeach data point appears only once, in order not to violate the inde-pendence assumptions necessary to compute the confidenceinterval.

5. Results

The results for the regression experiments, where we try to pre-dict the scores exactly, are presented in Table 1. These results arepresented separately for the different combinations of features,and for the overall acuity score and acuity scores for six nursingcare subsections. All the feature sets allow for significant outper-forming of the two simple baseline approaches. For all non-base-line results, the mean absolute error is below 0.5. First, weconsider the problem in which our aim is to predict the scoresfor tomorrow, in which case we only have access to the ‘‘PScore’’and ‘‘PText’’ features. Interestingly, the texts recorded for the pre-vious day prove to be more informative than the recorded scores,as in each case they enable more accurate predictions. These twoinformation sources complement each other in all cases; using

Table 1Regression results (mean absolute error). The regression errors for the different considered fpredicting the overall acuity score as well as for each of the six OPCq nursing care subsec

Features Acuity Subsec.1 Subsec.2

Previous day baseline 0.527 ± 0.003 0.518 ± 0.003 0.438 ± 0.003Majority voter baseline 0.715 ± 0.004 0.530 ± 0.003 0.491 ± 0.003PScore 0.491 ± 0.003 0.475 ± 0.003 0.415 ± 0.003PText 0.475 ± 0.003 0.471 ± 0.003 0.399 ± 0.003PScore & PText 0.443 ± 0.003 0.451 ± 0.003 0.373 ± 0.003Text 0.467 ± 0.003 0.452 ± 0.003 0.396 ± 0.003PScore & Text 0.432 ± 0.003 0.429 ± 0.003 0.367 ± 0.003

both the previous scores and the text gives much lower errors thanusing either of them separately. If we have access to the ‘‘Text’’ fea-tures recording the nursing notes for the same day on which thescore is to be given, the errors are even lower, but not dramaticallyso. Of the nursing care subsections, subsection 1 (planning and co-ordination of care) proves to be the most difficult to predict in eachcase, while subsection 2 (breathing, blood circulation and symp-toms of disease) proves to be the easiest to predict.

In Table 2, we present the C-index for each subsection (we donot consider the majority voter baseline here, since by definitionit would always result in a random concordance index of 0.5). Inall cases, the results are more accurate than the random perfor-mance of 0.5, demonstrating that the models do have predictivepower. Further, the models clearly outperform the previous daybaseline. The overall trends are the same as for the regressionexperiment - the text proves to be more informative than the pre-vious scores, combining them leads to the best models, and usingthe text from the same day allows for better prediction than thetexts recorded on the previous day. Subsection 1 is again the mostdifficult to predict, also in terms of the C-index. However, the eas-iest in this case is subsection 4 (personal hygiene and excretion).

6. Discussion

The aim of this research was to study to what degree the clinicalinformation in the EPRs of cardiac patients can be used to predicttheir OPCq acuity scores for the following day. Our hypothesiswas that textual nursing notes and previously assigned acuityscores can be utilized to predict the different subsections ofpatients’ acuity for the next day by applying machine learningtechniques. We tested our hypothesis by training and evaluatinga mathematical model to automatically assign patient acuityscores on a data set consisting of nursing documentation andrelated acuity scores from 23,528 electronic patient records. Themethods to predict a patient’s acuity were based on linguisticpre-processing, vector -space modeling of the text, and regularizedleast-squares regression.

The experimental results show that it is possible to accuratelypredict patients’ acuity scores. We considered settings in whichthe aim was to predict the scores exactly, as well as those wherethe aim was to rank patients in order from those needing the mostcare to those requiring the least. From the results, we can see anumber of interesting and consistent trends throughout all of theexperiments:

� The machine learning based approach to predicting acuityscores allows significantly better predictions than naïveapproaches such as predicting the score from the previousday, or using a majority score.� The texts and previous scores contain important and comple-

mentary information, with the best results being achieved bycombining both information sources.� Having access to notes from the same day that the prediction is

made further boosts predictive accuracy.

eature sets, as well as the two baseline methods. The errors are reported separately fortions (see Section 2 for description of the OPCq system).

Subsec.3 Subsec.4 Subsec.5 Subsec.6

0.454 ± 0.003 0.470 ± 0.003 0.477 ± 0.003 0.4463 ± 0.0030.598 ± 0.003 0.703 ± 0.003 0.748 ± 0.004 0.457 ± 0.0030.438 ± 0.003 0.449 ± 0.003 0.462 ± 0.003 0.428 ± 0.0030.433 ± 0.003 0.437 ± 0.003 0.437 ± 0.003 0.423 ± 0.0030.395 ± 0.003 0.406 ± 0.003 0.410 ± 0.003 0.400 ± 0.0030.427 ± 0.003 0.430 ± 0.003 0.430 ± 0.003 0.413 ± 0.0030.388 ± 0.003 0.394 ± 0.003 0.398 ± 0.003 0.390 ± 0.003

Table 2Ranking results (C-index%). The ranking accuracies for the different considered feature sets, as well as for the baseline method. The accuracies are reported separately forpredicting the overall acuity score, as well as for each of the six OPCq nursing care subsections (see Section 2 for description of the OPCq system).

Features Acuity Subsec.1 Subsec.2 Subsec.3 Subsec.4 Subsec.5 Subsec.6

Prev. day baseline 75.6 ± 0.3 65.5 ± 0.4 71.3 ± 0.3 73.8 ± 0.3 78.3 ± 0.3 76.8 ± 0.3 67.8 ± 0.3PScore 79.2 ± 0.4 68.2 ± 0.5 75.3 ± 0.4 77.2 ± 0.4 80.7 ± 0.4 79.5 ± 0.4 73.2 ± 0.4PText 79.9 ± 0.4 68.2 ± 0.5 75.9 ± 0.4 77.8 ± 0.4 82.6 ± 0.3 81.3 ± 0.4 71.8 ± 0.5PScore & PText 82.1 ± 0.4 70.2 ± 0.4 78.4 ± 0.4 80.5 ± 0.4 84.8 ± 0.3 83.6 ± 0.3 74.7 ± 0.4Text 80.8 ± 0.4 70.0 ± 0.4 76.8 ± 0.4 78.2 ± 0.4 83.3 ± 0.3 81.9 ± 0.3 73.1 ± 0.4PScore & Text 83.7 ± 0.3 72.6 ± 0.4 79.5 ± 0.4 81.5 ± 0.4 85.9 ± 0.3 84.7 ± 0.3 76.2 ± 0.4

E. Kontio et al. / Journal of Biomedical Informatics 51 (2014) 35–40 39

� Of the OPCq subsections, the predictions are the least accuratefor subcategory 1 (planning and co-ordination of care).

Currently, nurses define their patients’ acuity scores manuallyin the patient classification system once a day. This task is quitetime-consuming and a reliable patient acuity predicting systemwould thus decrease the nurses’ workload, freeing them for patientcare. The availability of this kind of predicting tool would alsotransform patient classification into a real-time process insteadof a monthly reporting task. Real-time data analysis and multi-useraccess to data also increases the ability to respond quickly andappropriately to changes in human resource allocation. The effec-tive use of patient classification provides improved resource utili-zation and the management of variances from planned resourceuse.

The possibilities to utilize free-form textual patient documenta-tion are currently weak. Our results suggest that there is significantpotential to improve this utilization. The models and tools used inthis study succeeded in using free-form textual patient documen-tation together with other information sources. Indeed, our find-ings show one example of the possible benefits of an integratedhospital information system. Thus, integrating information frompatient classification systems, human resource systems and elec-tronic patient record systems could support decision-making incardiac care. This could help organizations plan, manage, evaluateand support their tactical decision-making [41].

The predictive accuracy of the developed model was evaluatedusing cross-validation which demonstrated that the models wereindeed able to make accurate predictions. This suggests that thetrained mathematical models may be directly applicable for Finn-ish hospitals using the same kinds of electronic patient record sys-tems and recording practices, and treating similar types of patientsas those contained in our data. The results, however, cannot bedirectly generalized to other hospitals using different types ofrecording practices. In settings where the type of data differs sig-nificantly the trained models might either perform worse thanwas observed in our experiments, or not work at all if the differ-ences are substantial.

Here, we considered the problem of predicting acuity scoresbased on electronic patient records containing free-form nursingdocumentation and previous acuity scores. In these systems, othersources of information are also present, such as lists of medicationsadministered, diagnostic codes and notes recorded by doctors.Such information could be further leveraged to enable even moreaccurate predictions. These additional information sources com-bined with the positive results achieved with the available dataconfirm the need for an integrated hospital information system.Having even more accurate predictions would further support tac-tical decision-making, and could further improve the care process.

7. Conclusion

The results of this study confirm that it is possible to use elec-tronic textual nursing notes and previously assigned acuity scores

to predict a patient’s acuity for the next day through the applica-tion of machine learning techniques. In addition, there is roomfor a real-time integrated information system targeted for tacticaldecision-making.

Authors’ contributions

E.K. and S.S. designed the study. E.K., A.A., T.P., H.L.-L., T.S. andS.S. contributed to the data collection. The data was analyzed andinterpreted by A.A. and T.P. The other co-authors had the opportu-nity to comment on the statistical methods and analysis. E.K., A.A.and T.P. prepared the manuscript. H.L.-L., K.J., H.K., T.S. and S.S.contributed to revising and improving the manuscript.

Conflict of Interest

There were no conflicts of interest in the construction of thisstudy, or in the reporting of it.

Acknowledgments

The data was gathered as a part of larger research project ‘‘Clin-ical language management for improvement of utilization andusability of patient journals’’ funded by the Academy of Finland.We would like to thank Juho Heimonen for his help in preparingthe data, and Lingsoft Ltd for providing tools for languageprocessing.

References

[1] Fagerstrom L. Evidence-based human resource management: a study of nurseleaders’ resource allocation. J Nurs Manag 2009;17:415–25.

[2] Frilund M, Fagerstrom L. Managing the optimal workload by the PAONCILmethod – a challenge for nursing leadership in care of older people. J NursManag 2009;17:426–34.

[3] Heller FP, Drenth P, Koopman P, Rus V. Decisions in organizations: a threecountry comparative study. London: Sage; 1988.

[4] Winter AF, Ammenwerth E, Bott OJ, Brigl B, Buchauer A, Gräber S, et al.Strategic information management plans: the basis for systematic informationmanagement in hospitals. Int J Med Informatics 2001;64:99–109.

[5] Kontio E, Lundgren-Laine H, Kontio J, Korvenranta H, Salantera S. Informationutilization in tactical decision making of middle management healthmanagers. Comput Inform Nurs 2013;31:9–16.

[6] Hulshof PJ, Boucherie RJ, Hans EW, Hurink JL. Tactical resource allocation andelective patient admission planning in care processes. Health Care Manag Sci2013.

[7] Hulshof P, Boucherie R, Hans E, Hurink J. Tactical resource allocation andelective patient admission planning in care processes. Health Care Manag Sci2013.

[8] Dexter F, Ledolter J, Wachtel RE. Tactical decision making for selectiveexpansion of operating room resources incorporating financial criteria anduncertainty in subspecialties’ future workloads. Anesth Analg2005;100:1425–32.

[9] Dexter F, Traub RD. How to schedule elective surgical cases into specificoperating rooms to maximize the efficiency of use of operating room time.Anesth Analg 2002;94:933–42 [table of contents].

[10] Guerriero F, Guido R. Operational research in the management of the operatingtheatre: a survey. Health Care Manag Sci 2011;14:89–114.

[11] Testi A, Tanfani E. Tactical and operational decisions for operating roomplanning: efficiency and welfare implications. Health Care Manag Sci2009;12:363–73.

40 E. Kontio et al. / Journal of Biomedical Informatics 51 (2014) 35–40

[12] Cardoen B, Demeulemeester E, Beliën J. Operating room planning andscheduling: a literature review. Eur J Oper Res 2010;201:921–32.

[13] De Groot H. Patient classification system evaluation. Part 1: Essential systemelements. J Nurs Adminstr 1989;19:30–5.

[14] De Groot H. Patient classification system evaluation: Part Two, Systemselection and implementation. J Nurs Adminstr 1989;19:24–30.

[15] Huhkabay L, Skonieeczny R. Patient classification systems: the problems faced.Nurs Health Care 1981:89–103.

[16] Alward RR. Patient classification systems: the ideal vs. reality. J Nurs Adminstr1983;13:14–9.

[17] Giovannetti P. Staffing methods—implications for nurse staffing. In: Willis MLEL, editor. Measuring the quality of care Livingstone, London; 1984. p. 123–50.

[18] Fagerstrom L, Rainio AK, Rauhala A, Nojonen K. Validation of a new method forpatient classification, the Oulu Patient Classification. J Adv Nurs2000;31:481–90.

[19] Fagerstrom L. Benchmarking by the RAFAELA Patient Classification system – adescriptive study of optimal nursing intensity levels. Stud Health TechnolInform 2009;146:25–9.

[20] Fagerstrom L, Engberg IB. Measuring the unmeasurable: a caring scienceperspective on patient classification. J Nurs Manag 1998;6:165–72.

[21] Oulun yliopistollinen keskussairaala. Hoitoisuusluokituksen kehittäminenosaksi potilasryhmäkohtaista kustannuslaskentaa Oulun yliopistollisessakeskussairaalassa. Tutkimus- ja kehittämishankkeen loppuraportti. Oulu,Finland; 1994.

[22] Hoffman F. Nursing productivity assesment and costing out nursingservices. Philadelphia: J.B. Lippincott Company; 1989.

[23] Rainio A-K, Ohinmaa AE. Assessment of nursing management and utilizationof nursing resources with the RAFAELA patient classification system – casestudy from the general wards of one central hospital. J Clin Nurs2005;14:674–84.

[24] Kaustinen T. Hoitoisuusluokituksen kehittäminen ja arviointi. Oulu:Hoitotieteen laitos, Oulun yliopisto; 1995.

[25] Fagerstrom L. The patient’s perceived caring needs as a message of suffering. JAdv Nurs 1998;28:978–87.

[26] Fagerstrom L, Eriksson K, Engberg IB. The patient’s perceived caring needs:measuring the unmeasurable. Int J Nurs Pract 1999;5:199–208.

[27] Liljamo P, Kaakinen P. Cross-mapping the finnish classification of nursingdiagnosis, nursing interventions and the Oulu Patient Classification. StudHealth Technol Inform 2009;146:774–5.

[28] Hayrinen K, Saranto K, Nykanen P. Definition, structure, content, use andimpacts of electronic health records: a review of the research literature. Int JMed Inform 2008;77:291–304.

[29] Wang Z, Shah AD, Tate AR, Denaxas S, Shawe-Taylor J, Hemingway H.Extracting diagnoses and investigation results from unstructured text inelectronic health records by semi-supervised machine learning. PLoS ONE2012;7:e30412.

[30] Suominen H, Ginter F, Pyysalo S, Airola A, Pahikkala T, Salanterä S, et al.Machine learning to automate the assignment of diagnosis codes to free-textradiology reports: a method description. In: Schuurmans Dale, HauskrechtMilos, editors. Proceedings of the ICML/UAI/COLT 2008 workshop on machinelearning for health-care applications. Helsinki, Finland; 2008.

[31] Aronow D, Soderland S, Ponte J, Feng F, Croft W, Lehnert W. Automatedclassification of encounter notes in a computer based medical record. Medinfo1995:8–12.

[32] Hiissa M, Pahikkala T, Suominen H, Lehtikunnas T, Back B, Karsten H, et al.Towards automated classification of intensive care nursing narratives. Int JMed Informatics 2007;76. p. S362-S8.

[33] Ginter F, Suominen H, Pyysalo S, Salakoski T. Combining hidden Markovmodels and latent semantic analysis for topic segmentation and labeling:method and clinical application. Int J Med Informatics 2009;78. p. e1–6.

[34] Berman JJ. Confidentiality issues for medical data miners. Artif Intell Med2002;26:25–36.

[35] Raghavan Prabhakar DMC, Hinrich S. Introduction to informationretrieval. Cambridge University Press; 2008.

[36] Hoerl AE, Kennard RW. Ridge regression: biased estimation for nonorthogonalproblems. Technometrics 1970;12:55–67.

[37] Rifkin R, Yeo G, Poggio T. Regularized least-squaresclassification. Amsterdam: IOS Press; 2003.

[38] Harrell FE, Lee KLDBM. Tutorial in Biostatistics: multivariable prognosticmodels: issues in developing models, evaluating assumptions and adequacy,and measuring and reducing errors. Stat Med 1996;15:361–87.

[39] Gönen M, Heller G. Concordance probability and discriminatory power inproportional hazards regression. Biometrika 2005;92:965–70.

[40] Hanley JA, McNeil BJ. The meaning and use of the area under a receiveroperating characteristic (ROC) curve. Radiology 1982;143:29–36.

[41] Canela-Soler J, Elvira-Martinez D, Labordena-Barcelo MJ, Loyola-Elizondo E.Information systems in health and health indicators: an integratingperspective. Med Clin (Barc) 2010;134(Suppl 1):3–9.