However, the clinical implementation of well-designedcomputer-based structured checklists [2] and the diffusionof health information technology (HIT) are probably moresuccessful strategies to improve primary–secondary care
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l
mproving patient care [10]. Furthermore, a more widespreaddoption of electronic patient record systems by primary carehysicians (PCPs) seems to be linked to the principles of user-entered design [11].
Another strategy to improve performance by organizationss through feedback that individuals receive about their work12].
We assumed that applying the strategies highlighted in lit-rature (i.e. involving specialists and PCPs in working groupsith the aim to disseminate agreed guidelines with structured
eferral sheets, adopting easy user friendly electronic referralools, using feedback from secondary to primary care) mayead to an enhancement of the agreement between PCPs andpecialists’ priority assignments.
.2. Aims of the study
n the present study we describe the development of a clini-al priority approach to secondary care using a ‘progressivenvolvement scheme’ [13] of PCPs and specialists and wenalyze (i) the PCP-specialist agreement on classifying eachatient according to priority categories; (ii) the ability of theeedback and (iii) of an electronic referrals support systemERSS) to enhance this agreement.
We also hypothesize that ERSS may support the monitor-ng and assessing of any PCP’s prescription, with the aim ofbtaining more explicit clinical details for a more appropriatessignment of priorities.
The HIT system was developed and coordinated by thenformation Technology Service of the Local Health Unit (LHU)nd involved representative PCPs and specialists participatingn this study.
. Method
.1. Context
he study took place in a district of 85,000 inhabitants which isart of the LHU of the Autonomous Province of Trento, whichas almost 530,000 people. At the beginning (2000), 62 PCPs fordult patients were working in the district. To make a hospitaleferral in accordance with Italian law, PCPs have to complete
standard form, which includes clinical details of the patientnd the diagnostic procedures requested. Today almost allCPs use electronic health records for their daily activity inhe clinic.
.2. Development of a clinical priority approach
he clinical prioritization approach called ‘Homogeneousaiting Groups’ (HWGs) was developed by a steering com-ittee set up in 2000. One of the authors (GM), medical
irector of the district hospital assigned by the managementoard of the LHU, aimed to build consensus between PCPsnd specialists [14,15] using a ‘plan-do-check-act’ approach
nd a ‘progressive involvement scheme’ [13] by setting up
steering committee which includes representative PCPsnd specialists, and the head of the appointment book-ng center. The steering committee identified 5 categories
o r m a t i c s 8 2 ( 2 0 1 3 ) 1144–1151 1145
of diagnostic examinations (outpatient) as most problem-atic for appropriate access and extended waiting times: (i)ophthalmology referral, (ii) neurological and orthopedic CTscans, (iii) neurological and orthopedic MR scans, (iv) abdom-inal ultrasounds, and (v) upper-aortic and peripheral Dopplerobservations.
The steering committee also identified and coded 5 clinicalcategories of homogeneous waiting times:
– A (maximum waiting time of 3 days),– B (not more than 10 days),– C (not more than 30 days),– E or NO LETTER (without a maximum wait),– P (planned follow-up examination).
For urgent referrals, patients had to attend the Accidentand Emergency Department (A&E) of the hospital with thestandard referral form marked ‘urgent’. Each category wasdefined on the assumption that the waiting time for diagno-sis, between PCP referral and specialist examination, wouldnot impair the patient’s prognosis [14].
It is important to emphasize that patients who do notreceive a letter are considered without priority, although thePCP has the option of specifying this explicitly using the letterE. This means that PCPs are not required to fill in a referralform if the patient is not considered to belong to any one ofthe priority categories.
The waiting time categories were similar to those describedby the Access Criteria For First Specialist Assessment (FSA),developed in New Zealand in the late 90s for some diagnosticprocedures [16].
Initially (in 2000), only 20 PCPs were involved out of 62 fromthe district, selected according to their willingness to partici-pate in the study. For each referral they were asked to assign 1of the 5 HWG categories identified by the steering committee.On the basis of this information, the booking center workingwith HIT tools assigned the patient a priority correspondingto the HWG.
The scheme was then extended to all PCPs in the pilotdistrict, and also to diagnostic examinations not includedin the pilot study. The final operating manual, paper-basedand pdf containing referral instructions and guidelines for 84diagnostic examinations and procedures each with the samestandardized waiting categories, was distributed to PCPs andspecialists [17]. After about three years, all PCPs and special-ists across the LHU of the Autonomous Province of Trento havebegun to use HWGs.
Tables 1 and 2 provide an example of the HWG proce-dure applied to orthopedic and Ear, Nose and Throat (ENT)examinations, two of the most common referrals. All work-ing subgroups agreed to add to almost all priority categoriesthe keyword “Other (10%)” after the clinical keywords (suchas explicit signs, symptoms and suspected pathologies), withthe aim to include any condition not fitting with the explicit
keywords. The function of the “10%” would be “inhibitory”,i.e. reminding PCPs not to use priority categories A, B andC for clinical conditions not indicated by the explicit key-words.
1146 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1144–1151
Table 1 – Clinical “keywords” for each priority level of orthopedic examination.
Code onprescription form
Maximum wait Indications or clinical “keywords”
Urgent A&E Department – Conditions to refer by PCP immediately or in less than 24 hHWG A 3 days – Large joint mechanical lock (Shoulder-Elbow-Hip-Knee) with movement
limitations– Acute sciatica with inability to maintain erect posture– Acute neck pain– Acute knee pain without injury (surgery is recommended)– Other (10%)
HWG B 10 days – Treatment resistant acute diseases (e.g. epicondylitis, periarthritis, whenrheumatic disorders of the wrist and hand are suspected)– Malignant or suspected malignant diseases (e.g. osteolysis)– No chronic joint effusion– Other (10%)
HWG C 30 days – Canalicular syndrome (e.g. carpal tunnel, trigger finger)– Sub-acute disease– Chronic or sub-acute knee pain (surgery is recommended)– Other (10%)
HWG E No limit – Other clinical conditionsHWG P Follow up time – Check on post-discharge and other follow up
HWG = homogeneous waiting group.Other (10%) = other clinical indications, only in few cases!
2.3. Concordance between PCPs and specialists
The steering committee also encouraged specialists to assignone of the priority categories to each referred patient at the
time of examination in order to better assess the degree ofagreement between PCPs and specialists in assigning prioritylevels.
Table 2 – Clinical “keywords” for each priority level of Ear, Nose
Code onprescription form
Maximum wait
Urgent A&E Department – ConditiHWG A 3 days – Recurri
– Few day– Odynop– Other (1
HWG B 10 days – More th– Ear pain– Small b– Recurri– Salivary– Acute e– Adenot– Varioushead or neyelid pto– Not inv– Other (1
HWG C 30 days – Persiste– Other (1
HWG E No limit – Adenot– Oral Lic– Other (1
HWG P Follow up time – Follow-
HWG, homogeneous waiting group.Other (10%) = other clinical indications, only in few cases!OSAS, Obstructive Sleep Apnea Syndrome.
2.4. Health information technology
With the introduction of more sophisticated HIT in our LHU in2008, PCPs began receiving electronic feedback from special-
ists for each examination performed. This included outpatientdischarge summary and priority category reassigned by spe-cialists.
and Throat (ENT) examination.
Indications or clinical “keywords”
ons to refer by PCP immediately or in less than 24 hng epistaxis in the past 48 hs onset neck masshagia0%)an 2 weeks dysphagia and/or dysphonia
resistant to antibiotic therapyleeding from upper respiratory tractng epistaxis
gland swellingxacerbation of Ménière’s syndromeonsillar hypertrophy and OSAS in children
ENT signs and/or symptoms of suspected neoplastic etiology (e.g.eck mass, facial paresis, monolateral rhinostenosis, exophthalmos,sis, diplopia)
alidating dizziness0%)nt adenotonsillar hypertrophy in children0%)
i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1144–1151 1147
c
2
2W62urstbctraa
2WPsbttstss
2IuTksFclrruqdrr13
wct
2Tocs
Fig. 1 – Example of the interface allowing a PCP to place arequest to refer a patient to a brain CT-scan examination.The PCP can select clinical keywords from the prioritycategories (right column) and in a subsequent step the PCP
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l
In 2010, we developed an ERSS which included clinical indi-ations to support PCPs in choosing priority categories.
.5. Comparison of three referral phases
.5.1. First phase (2001–2007)e used the data gathered by the referrals generated by the
2 PCPs and then reassigned to HWGs by specialists during004 (group 1 – 1126 records). In this phase all PCPs had tose the paper-based or pdf manual. All the priority categories,eassigned and reported on paper by the specialists, were sub-equently recorded by an administrative team of the LHU andhen stored in the same software used by the appointmentooking center. Every six months a paper-based communi-ation was sent to all PCPs to show: (1) the referral rate ofhe absolute number of HWGs prescribed by each PCP withespect to the median value of all PCPs; (2) the single percentgreement value of PCP-specialist versus the median percentgreement value of all PCPs.
.5.2. Second phase (2008–2011)e used the referrals prescribed during 2011 by 32 out of 62
CPs who did not use ERSS, and then reassigned to HWGs bypecialists (group 2 – 597 records). In this phase specialistsegan to use a computer-based tool to complete the outpa-ient discharge summary and reassigned the priority categoryo each patient, incorporating this evaluation in the dischargeummary. This meant that as all PCPs had been using an elec-ronic health record linked with specialists’ record-keepingystems, they could promptly view the outpatient dischargeummary and priority category reassigned by the specialists.
.5.3. Third phase (2010–2011)n this phase the manual was incorporated in the softwaresed by many PCPs and it was used daily by 30 out of 62 PCPs.he ERSS allows all PCPs to visualize the list of the clinicaleywords as they appear in the HWG and so choose signs,ymptoms or suspected pathologies related to the patient –ig. 1 shows an interface of the electronic prescription (in thisase a PCP chooses clinical “keywords” in the priority rankingist, to refer a patient for a brain CT-scan). We used the refer-als prescribed during the first six months of 2011 and theneassigned to HWGs by specialists. To highlight the effects ofsing ERSS, we selected the data of the PCPs who most fre-uently used the system, and for this reason, we gathered theata related to those PCPs who explained about 85% of refer-als. In this way, we selected 14 out of 30 PCPs (the range of alleferrals prescribed by ERSS in the first six months of 2011 by4 PCPs is 832-3061, and by the other 16 PCPs is 42-602). Group
is made up of 639 records.To investigate if the concordance of the 14 PCPs in 2004
as different from that of the remaining PCPs of group 1, wealculated the simple kappa of the former PCPs compared withhe simple kappa of the latter in the same year.
.5.4. Data collected
he following data were collected and analyzed: (i) categoryf referrals (ENT and orthopedic examinations), (ii) identifi-ation number of each PCP, (iii) identification number of eachpecialist, (iv) HWG reported by PCPs for each patient, (v) HWG
enters some free clinical details about the patient’s case.
reported by specialists for each patient, (vi) clinical data of thepatient in each prescription form of group 3.
2.5.5. Statistical analysisThe degree of agreement between the priority assigned byPCPs and specialists can be assessed using a simple kappastatistical analysis [18]. We used SAS 9.1.3 (SAS Institute Inc.).The statistical agreement value was called “concordance”.Proportions were compared using the �2 test. All tests weretwo-tailed, and a p value of less than 0.05 was consideredstatistically significant.
2.6. Clinical “keywords” evaluation
Using the PCPs’ prescription database, we described the rela-tive frequency of each “clinical keyword” of each HWG selectedby PCPs using ERSS (group 3). Considering that PCPs can addfree text details about the patient’s case, we wanted to studythe effectiveness of the “inhibitory” function of “(10%)”, whilesearching any additional keywords.
3. Results
The first group of 62 PCPs provided 1126 records of referrals;the second group corresponding to the subset of referrals asso-ciated to the 32 PCPs who did not utilize ERSS provided 597records; the third group corresponding to the subset of refer-rals related to the 14 PCPs using ERSS provided 639 records.
3.1. Concordance analysis
Table 3 shows the HWG categories assigned by the 62 PCPs,
against those assigned by specialists during 2004 (group 1).
Table 4 shows the HWG categories assigned referring activ-ity during 2011 by 32 PCPs who received feedback through
(60 out of 91). On the other hand, we observed a relatively highfrequency of strings such as “hearing loss” (36 out of 99) in ENTreferrals.
0
0,2
0,4
0,6
0,8
1
1,2
THIRD PHASE (14 PCPs)SECOND PHASE (32 PCPs)FIRST PHASE (62 PCPs)
SIM
PLE
K VA
LUE
Total 116
electronic discharge summary but did not use ERSS, againstthose assigned by specialists (group 2).
Table 5 shows the HWG categories assigned during thefirst six months of 2011 referring activity by the14 PCPs whoreceived feedback through electronic discharge summary andwho use ERSS, against those assigned by specialists (group 3).
In group 1, the simple kappa statistic was 0.564 (moder-ate concordance), with a 95% CI of 0.526–0.602; in group 2, thesimple kappa statistic was 0.668 (a value of substantial con-cordance) with a 95% CI of 0.619–0.716; whereas in group 3using ERSS there was an almost perfect degree of agreement,with a simple kappa statistic of 0.883 (very high concordance)and a 95% CI of 0.854–0.912.
The simple kappa statistics of the 14 PCPs in 2004 (180records) was equal to 0.614 (95% CI 0.518–0.709); the simplekappa of the other 48 out of 62 PCPs in 2004 (1126 − 180 = 946records) was 0.554 (95% CI 0.512–0.596). The test for equalkappa coefficients was not statistically significant betweenthe two groups (�2 = 1.268, 1 df, p = 0.260). We could thereforeinfer that the 14 PCPs had the same referral attitude of theremaining PCPs in 2004.
There was a significant difference in the proportion of con-cordance among the three groups (�2 = 182.5, 2 df, p < .0001)(Fig. 2).
The analysis of the proportion of priority referrals used byPCPs (i.e. HWGs A, B, C, E, P) throughout the three phasesshows a significant statistical difference (�2 = 782.2, 14 df,p < .00001). A significant statistical difference is also observedbetween the first two phases (�2 = 21.8, 9 df, p < .01) (Table 6).Furthermore, the proportion of priority referrals used by thethird group (14 PCPs) in 2011 shows a significant differ-
ence compared to the proportion of priority referrals usedby the same PCP group in 2004 (�2 = 153.8, 9 df, p < .00001)(Table 7).
Table 4 – Priority categories assigned by 32 PCPs (horizontal) no– group 2 (year 2011).
Prior
A
Priority categories assigned by PCPs A 92
B 5
C 0
E 0
P 0
Total 97
490 408 76 36 1126
3.2. “Keywords” description
The analysis of HWG keywords selected by the 14 PCPs usingERSS affects 236 ENT and 403 orthopedic examinations. A rel-atively high prevalence of follow-ups was observed in eachreferral examination – 56 out of 236 in ENT and 123 out of 403in orthopedic examinations. Furthermore, we observed a highfrequency of the heading “Other (10%)” for each referral – 99out of 236 in ENT and 91 out of 403 in orthopedic examina-tions. Searching for more repetitive strings in the specific fieldof the PCPs’ prescription database, we observed a relativelyhigh frequency of “alg” and “dol” in orthopedic referrals (cor-responding to the roots of the Italian words “ache” and “pain”),which represented 66% of the overall heading of “Other (10%)”
Fig. 2 – The agreement of PCPs versus specialists priorityassignment is increasing between the three groups(�2 = 182.5, 2 df, p < .0001).
t using ERSS against assignments by specialists (vertical)
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1144–1151 1149
Table 5 – Priority categories assigned by 14 PCPs (horizontal) using ERSS against assignments by specialists (vertical) –group 3 (first six months of 2011).
.1. Concordance of PCPs-specialists and HWGroportion rates
mproving the referral process through the use of protocolsnd other forms of close interaction between primary andecondary care is illustrated in literature [1,19] and is recom-ended in institutional plans [20]. A systematic review reports
n improvement in the performance when clinicians imple-ent clinical decision support systems tailored to (a) provide
ecision support automatically as part of clinician workflow,b) deliver decision support at the time and location of the deci-ion making, (c) provide actionable recommendations, and (d)se a computer-based tool to generate the decision support
3].We observed a moderate level of concordance in the first
hase of the primary–secondary care interface, but then thegreement significantly increases, showing a substantial con-ordance in the second phase and a very high concordance in
he third. The global rise in the referral activity in 2011 fromCPs using ERSS (third phase) is explained by the tendencyf the same PCPs to increase prescriptions, but especially by
Table 7 – Priority categories used by the 14 PCPs of the third grophase: with ERSS).
First phase(2004)
Nr.
Priority categories assigned by 14 PCPsof the third group
A 29
B 91
C 56
E 1
P 3
Total 180
8 1.3 128 20.0 175 7.49 1.5 179 28.0 202 8.6
597 100.0 639 100.0 2362 100.0
the utilization of HWG E (no maximum waiting time) andP (planned repeat examination for follow-up). A contextualdecreasing of higher priority categories (HWG A and HWG B)was also observed.
4.1.1. Weaknesses and strengthsThe main weakness of the “concordance” assessment methodwere: (i) patients were examined by PCPs and by specialistsat different times, and in several cases delayed by days; (ii)the reassigning of the priority category was not performed byspecialists unaware of the study, even though they had no wayof knowing if the PCP had used ERSS or not; and (iii) specialistsreassigned the HWG only to referrals booked with a priorityletter on the standard form and not to referrals booked withoutany priority (explained by an almost absent discordance inreferrals without priority).
We can therefore suggest that this study highlights threetools for improving primary care-specialist referrals process:(i) the involvement of PCPs and specialists in the creation of
a manual for referrals that emphasizes the effectiveness ofan integrated PCP-specialist working approach; (ii) the promptfeedback received by PCPs from specialists regarding patient’sclinical priority category, that seems to be associated with an
up in 2004 (first phase: without ERSS) and in 2011 (third
c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1144–1151
Summary pointsWhat was known before the study?
• Literature highlights that effective strategies toenhance the referring process generally include: (i)dissemination of guidelines with structured referralsheets and (ii) involvement of specialists in educa-tional activities.
• Literature highlights that an important variable forimproving performance in an organization is the feed-back that individuals receive.
• The improvement of health information technology atprimary and secondary care interface and the clin-ical usability of well-designed computer-based toolsare probably successful strategies to enhance the dis-semination of structured paper-based protocol forimproving referral appropriateness in the daily activityof PCPs and specialists.
What has this study added to the body of knowledge?
• The involvement of PCPs and specialists in the creationof a manual for referral activity under explicit clinicalindications and according to access criteria based onclinical priorities emphasizes the effectiveness of anintegrated PCP-specialist working approach.
• The electronic feedback received by PCPs from special-ists regarding clinical reports and assessment of thepatient’s clinical priority category appears to be linkedto an increase in PCP-specialist agreement.
• The dissemination of a manual to provide automaticdecision support as part of the PCP workflow results in
r
1150 i n t e r n a t i o n a l j o u r n a l o f m e d i
increase in concordance; and (iii) the dissemination of a man-ual to provide automatic decision support as part of the PCPworkflow that results in an even greater agreement with spe-cialists and in a greater difference of priority categories used.
4.2. Clinical “keywords”
In the work by Gandhi et al. [21], it is underlined that electronicreferrals can facilitate the inclusion of decision support intothe referral process. In our experience, we did not use an auto-matically visible transmission of clinical words by means ofan electronic tool from PCPs to specialists, but only from spe-cialists to PCPs. Indeed, our tool helps PCPs to select clinical“keywords” that are closer to patient characteristics, select-ing at the same time maximum acceptable waiting times tocarry out an examination. For example, an ENT consultationfor a patient with “more than 2 weeks dysphagia and/or dys-phonia” requested within a time frame of not more than10days, might lead a PCP to a correct and timely referral and,at the same time, it may prevent unknown and unnecessaryreferrals.
4.2.1. Weaknesses and strengthsOur study suggests that an ERSS based manual did not demon-strate the ability to describe most of the clinical keywords usedin practice by PCPs, as 99 out of 236 ENT referrals and 123out of 403 orthopedic referrals were explained by the keyword“Other”. This demonstrates that the “10%” value has not hadany “inhibitory” effect. However, ERSS was able to identify themost frequent “unknown” motivations of ENT and orthopedicreferrals, which were not identified at the beginning by theworking subgroups.
5. Conclusion
We tried to match the daily practice of PCPs and specialists inattributing clinical priority categories to each referral patient,but we know that this is not an adequate setting for a con-trolled study, and moreover unmeasured confounders couldaccount for some differences we observed. In any case, it isworth noting that a strong impact on the daily activity of PCPsmight come from (i) agreement between PCPs and specialistson protocols and guidelines, (ii) prompt feedback on clinicaldetails and (iii) from ERSS able to promptly guide decisionmaking throughout the clinical decision process facilitatingcontinuous assessment of the content of clinical protocols.
Authors contributions
G.M. managed the diffusion of the HWG model in the LHU dis-trict from 2001 to present, designed the study, participated indata analysis, drafted the manuscript; M.G. analyzed the data;V.D. extracted data from the data warehouses. All authors readand approved the final manuscript.
Conflict of interest
The authors have no financial interest to this work.
increased agreement with specialists.
Acknowledgements
The authors wish to thank all PCPs and hospital specialistsof the LHU district who actively participated in the imple-mentation and dissemination of the model based on clinicalpriorities and PCPs who significantly utilize ERSS. The authorsthank Nivedita Agarwal (Radiology Department) and LeonardoSartori (Information Technology Service) for editing this paper.A particular thanks to Rita Fellini (Medical Direction), foradministrative assistance and data presentation.
e f e r e n c e s
[1] A. Akbari, A. Mayhew, M.A. Al-Alawi, J.M. Grimshaw, R.Winkens, E. Glidewell, et al., Interventions to improveoutpatient referrals from primary care to secondary care,CDS Rev. (October (4)) (2008),http://dx.doi.org/10.1002/14651858.CD005471, Art. No.:CD005471.
[2] L.Y. Liu, J.C. Wyatt, J.J. Deeks, S. Clamp, J. Keen, P. Verde,et al., Systematic review of clinical decision tools for acuteabdominal pain, Health Technol. Assess. 10 (47) (2006) 1–167(iii–iv).
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l
[3] K. Kawamoto, C.A. Houlihan, E.A. Balas, D.F. Lobach,Improving clinical practice using clinical decision supportsystems: a systematic review of trials to identify featurescritical to success, BMJ 330 (2005) 765–772.
[4] C.M. DesRoches, E.G. Campbell, R.R. Sowmya, K. Donelan,T.G. Ferris, A. Jha, et al., Electronic health records inambulatory care – a national survey of physicians, N. Engl. J.Med. 359 (1) (2008) 50–60.
[5] M.J. Armstrong, C. Booth, Electronic health records inambulatory care, N. Engl. J. Med. 359 (17) (2008) 1848–1849.
[6] S.J. McPhee, B. Lo, G.J. Saika, R. Meltzer, How good iscommunication between primary care physicians andsubspecialty consultants? Arch. Intern. Med. 144 (6) (1984)1265–1268.
[7] T.K. Gandhi, D.F. Sittig, M. Franklin, A.J. Sussman, D.G.Fairchild, D.W. Bates, Communication breakdown in theoutpatient referral process, J. Gen. Intern. Med. 15 (9) (2000)626–631.
[8] A.H. Chen, H.F. Yee Jr., Improving the primary-specialty careinterface: getting from here to there, Arch. Intern. Med. 169(11) (2009) 1024–1026.
[9] Y. Kim, A.H. Chen, E. Keith, H.F. Yee Jr., M.B. Kushel, Notperfect, but better: primary care providers’ experiences withelectronic referrals in a safety net health system, J. Gen.Intern. Med. 24 (5) (2009) 614–619.
[10] E.L. Abramson, V. Patel, S. Malhotra, E.R. Pfoh, S.N. Osorio, A.Cheriff, C.L. Cole, A. Bunce, J. Ash, R. Kaushal, Physicianexperiences transitioning between an older versus newerelectronic health record for electronic prescribing, Int. J.
Med. Inf. 81 (2012) 539–548.
[11] T. Christensen, A. Faxvaag, Lœrum Hallvard, A. Grimsmo,Norwegians GPs’ use of electronic patient record systems,Int. J. Med. Inf. 78 (2009) 808–814.
o r m a t i c s 8 2 ( 2 0 1 3 ) 1144–1151 1151
[12] J.R. Folkman, The Power of Feedback, John Wiley & Sons,2006.
[13] P. Hersey, K. Blanchard, Management of OrganizationalBehavior, Prentice-Hall, Inc., Englewood Cliffs, NJ,1982.
[14] G. Mariotti, Priorità cliniche in sanità [Clinical Priority inHealth Care], Franco Angeli, Milan, 2006.
[15] G. Mariotti, R. Sommadossi, T. Langiano, R. Raggi, Italy’spublic health system is changing from waiting times topriority [Letter], BMJ 318 (1999) 1698–1699.
[16] Ministry of Health, New Zealand, Manatu Hauora,http://www.health.govt.nz/our-work/hospitals-and-specialist-care/elective-services (accessed 28.02.13).
[17] APSS (Azienda Provinciale per i Servizi Sanitari), ManualeRAO, http://www.apss.tn.it/Public/allegati/DOC 648724 0.pdf(accessed 28.02.13).
[18] J.R. Landis, G.G. Koch, The measurement of observeragreement for categorical data, Biometrics 33 (1) (1977)159–174.
[19] E.G. Poon, T.K. Gandhi, T.D. Sequist, H.J. Murff, A.S. Karson,D.W. Bates, I wish I had seen this test result earlier!, Arch.Intern. Med. 164 (Nov 8) (2004) 2223–2228.
[21] T.K. Gandhi, N.L. Keating, M. Ditmore, D. Kiernan, R. Johnson,E. Burdick, et al., Improving referral communication using areferral tool within an electronic medical record, in: K.Henriksen, J.B. Battles, M.A. Keyes, M.L. Grady (Eds.),
Advances in Patient Safety: New Directions and AlternativeApproaches (Vol. 3: Performance and Tools), Agency forHealthcare Research and Quality (US), Rockville, MD,2008.
