31
Emergency Care Institute Website Effectiveness Project FINAL REPORT
EmergencyCareInstitute
WebsiteEffectivenessProject
FINALREPORT
Page 2/31
Acknowledgements
The author thanks Matthew Murray for obtaining the HIE data on annual ED presentations, and providing data from the Internet Access Survey.
AGENCY FOR CLINICAL INNOVATION Level 4, Sage Building 67 Albert Avenue Chatswood NSW 2067 Agency for Clinical Innovation PO Box 699 Chatswood NSW 2057 T +61 2 9464 4666 | F +61 2 9464 4728 E [email protected] | www.aci.health.nsw.gov.au Author: Dr Jeremy Lawrence, Advanced Emergency Medicine Trainee, Emergency Care Institute Supervisor: A/Prof Sally McCarthy, Clinical Director, Emergency Care Institute SHPN: (ACI) 150090 ISBN: 978-1-76000-149-0 Phone. +61 2 9464 4674 Email. [email protected] Further copies of this publication can be obtained from: Agency for Clinical Innovation website at: www.aci.health.nsw.gov.au Disclaimer: Content within this publication was accurate at the time of publication. This work is copyright. It may be reproduced in whole or part for study or training purposes subject to the inclusion of an acknowledgment of the source. It may not be reproduced for commercial usage or sale. Reproduction for purposes other than those indicated above, requires written permission from the Agency for Clinical Innovation. © Agency for Clinical Innovation 2015
Page 3/31
Table of Contents
1. EXECUTIVE SUMMARY 5
1.1 About the ECI and its website resources 5
1.2 The goal of this project is to increase the effectiveness of the ECI website 5
1.3 Phase 1: Analyse website usage 5
1.4 Phase 2: Identify impediments to use of the website 6
Awareness of the ECI website is fundamental 6
Poor Internet access impedes access to the ECI website in the Northern NSW LHD 6
Usability of the ECI website is generally good 6
1.5 Phase 3: Recommended actions 6
2. INTRODUCTION 8
2.1 About the ECI 8
2.2 The ECI website publishes resources of use to ED clinicians in NSW 8
2.3 The goal is to increase the effectiveness of the ECI website 8
2.4 Prior Research 9
2.4.1 ECI Review of Internet Access and Usage in Emergency Departments 9
2.4.2 Broader Research on Website Usage 10
3. PHASE 1: ANALYSE WEBSITE USAGE 13
3.1 Aim 13
3.2 Method 13
3.2.1 The ECI website uses Google Analytics to collect data on its use 13
3.2.2 Google Analytics identifies the city or town of each website user 13
3.2.3 eHealth NSW provided additional information to identify users by LHD 13
3.2.4 Google Analytics was configured to identify use by each LHD 14
3.2.5 The number of website sessions was compared to the number of ED presentations 14
3.3 Results 15
3.3.1 Details & Discussion 15
3.3.2 Summary 16
4. PHASE 2: IDENTIFY IMPEDIMENTS TO USE OF THE WEBSITE 17
4.1 Aim 17
4.2 Method 17
4.2.1 Information was sought from ED staff 17
4.2.2 Questionnaire 18
4.3 Results 18
4.3.1 Responses 18
4.3.2 Factors encouraging use of the ECI website 18
4.3.3 Factors impeding use of the ECI website 19
4.3.4 Internet access 21
4.3.5 Usability of the ECI Website 22
4.3.6 The usefulness of the ECI website 22
4.4 Analysis and Summary of Findings 23
Page 4/31
4.4.1 Awareness of the ECI website is fundamental 23
4.4.2 Poor Internet access impedes access to the ECI website in the Northern NSW LHD 24
4.4.3 Usability of the ECI website is generally good 24
4.4.4 Results from this phase are consistent with the Phase 1 results 24
5. PHASE 3: RECOMMENDED ACTIONS 25
5.1 Raising awareness of the ECI website 25
5.1.1 Raising awareness amongst permanent staff 25
5.1.2 Raising awareness amongst junior and locum staff 25
5.2 Internet access difficulties 26
5.3 ECI website usability 27
5.3.1 General usability 27
5.3.2 Broken links 27
6. SUMMARY & CONCLUSIONS 28
6.1 Phase 1 28
6.2 Phase 2 28
Awareness of the ECI website is fundamental 28
Usability of the ECI website is generally good 28
6.3 Phase 3 29
7. APPENDIX: PHASE 2 QUESTIONAIRE 30
7.1 Description & Purpose 30
7.2 Questions 30
Page 5/31
1. Executive Summary
1.1 About the ECI and its website resources The Emergency Care Institute (ECI) has been set up under the umbrella of the Agency for Clinical Innovation with its primary role being to improve outcomes for patients presenting at hospital Emergency Departments (EDs). A major part of the ECI’s role is producing resources of use to clinicians and patients in EDs. The resources include:
Clinical tools - including guidelines, procedures, and forms for use by clinicians Patient factsheets - designed to be given to patients at discharge
These resources are published on the ECI website www.ecinsw.com.au.
1.2 The goal of this project is to increase the effectiveness of the ECI website
The main goal of this project is to increase the effectiveness of the ECI website in providing resources of use to ED clinicians and patients in NSW. This is being achieved in three phases, which are described in turn.
1.3 Phase 1: Analyse website usage This phase has identified Local Health Districts (LHDs) which are heavy and light users of the ECI website. The method was based on use of the Google Analytics website analysis tool in combination with additional information provided by eHealth NSW. The information was sufficient to identify usage from 9 of the 15 LHDs in NSW, and one particular hospital, as shown in the following table. Although the data does not cover all LHDs in NSW, it is sufficient for the purposes of this project as it clearly identifies relatively heavy and light users of the ECI website.
ECI website use by LHD: Summary
Note: IP address data has some limitations. It was only possible to reliably identify users by LHD, not by individual hospitals (with the exception of Royal Prince Alfred Hospital as indicated above).
2.56
9.73
14.05
14.93
17.13
20.86
78.76
0.00 20.00 40.00 60.00 80.00
Mid North Coast & Northern NSW
Far West & Western NSW
Hunter New England
Murrumbidgee
Southern NSW
Illawarra Shoalhaven & South Eastern…
Royal Prince Alfred Hospital
Website sessions per 1,000 patient presentations
Page 6/31
1.4 Phase 2: Identify impediments to use of the website This phase collected qualitative data on impediments to use of the ECI website, by focussing on the LHDs which are the lowest users of the website, namely Mid North Coast LHD and Northern NSW LHD. A questionnaire was administered by phone interview with staff from the six largest EDs in the districts, representing 62% of ED presentations, and also to staff from a selection of medium-sized and smaller EDs. Results allowed several factors to be identified consistently across several responses. Factors were identified as follows:
Awareness of the ECI website is fundamental
o There is good awareness of the ECI itself, but that does not generally extend to frequent use of the ECI’s resources. Where the ECI website is most widely known, it is known by virtue of its use: certain ECI resources are frequently used, and the ECI website becomes visible to staff because they see it used. There was agreement that better awareness of the ECI website would make it more useful.
o The ECI is primarily seen as a source for patient factsheets. There is substantially less awareness of clinical tools and other resources.
o Several respondents identified a particular lack of awareness of the ECI website amongst more junior and locum staff compared to permanent staff.
Poor Internet access impedes access to the ECI website in the Northern NSW LHD
o In the Northern NSW LHD, the speed and reliability of Internet access was identified as specific issue: access to the Internet is slower and sometimes less reliable than access to intranet sites, and specific resources linked from the ECI website, notably videos, could not be accessed.
Usability of the ECI website is generally good
o Usability of the website was rated highly by respondents, with only a minority having difficulties navigating the website.
o A specific technical deficit was discovered with respect to broken links on the ECI website.
1.5 Phase 3: Recommended actions The above factors were analysed and recommendations made to promote use of the ECI website. Recommendations are:
1. The ECI send out regular, e.g. quarterly, emails on ‘what is new on the ECI website’.
2. Resources on the ECI website be demonstrated routinely at ECI symposia and fora.
3. (Possible) The ECI provide minor computer equipment of practical use to EDs, such as mousepads or mice, printed with ECI branding and messages.
4. (Possible) The ECI produce a flyer for inclusion in staff orientation packs introducing the ECI resources, and locate and contact the relevant staff to enable this to happen.
5. (Possible) The ECI produce a slide-set or video introducing the ECI resources for use at regular staff teaching sessions, and contact the relevant staff to encourage its use.
Page 7/31
6. (Possible) The ECI identify material on the website which is useful to prevocational doctors on wards outside the ED, and circulate this material by email.
7. (Possible) The ECI contact the Chief Information officer of the Northern NSW LHD to provide feedback on the Internet access difficulties reported by respondents there.
8. (Possible) Following a forthcoming ECI website redesign, opinions could be sought again on usability, either by a questionnaire or by a discussion at a symposium or forum.
9. As part of the ECI website redesign, the mechanism for detecting broken links should be reviewed, to ensure that it detects redirected links as well as completely broken ones.
The recommendations marked as “possible” are of uncertain benefit or are otherwise challenging. Most relate to raising awareness of the ECI amongst locum and more junior staff, which is a challenging problem for reasons which are analysed in the report.
At the time of writing, the ECI is considering the recommended actions and will complete Phase 3 of the project by implementing suitable ones.
Page 8/31
2. Introduction
2.1 About the ECI The Emergency Care Institute (ECI) has been set up under the umbrella of the Agency for Clinical Innovation with its primary role being to improve outcomes for patients presenting at hospital Emergency Departments (EDs) across NSW through coordination, networking and research.
The ECI works with and supports (Emergency Department) ED staff, in consultation with consumers and the community, to research, plan and deliver more effective and efficient care leading to better outcomes for patients. It provides a strong, independent voice for improvement on behalf of staff, patients and NSW communities. Specific priorities for the ECI include:
Reduce variation in clinical practice Provide a forum for the systematic analysis and assessment of information regarding
the quality of care in NSW EDs Display strong leadership to inform the future strategic direction of emergency care in
NSW Undertake audits of activities to ensure that there are meaningful and transparent
results, which are widely available.
2.2 The ECI website publishes resources of use to ED clinicians in NSW A major part of the ECI’s role is producing resources of use to clinicians and patients at NSW EDs. The resources include:
Clinical tools, which are guidelines, procedures, and forms of use to clinicians in particular clinical circumstances
Patient factsheets designed to be given to patients by clinicians in particular clinical circumstances
These resources are published on the ECI website www.ecinsw.com.au.
2.3 The goal is to increase the effectiveness of the ECI website The goal of this project is to increase the effectiveness of the ECI website in providing resources of use to ED clinicians in NSW. This is being achieved in three phases:
Phase 1: analysis of the comparative use of the ECI website by various EDs in NSW Phase 2: analysis of impediments to use of the ECI website by the sites using it least Phase 3: developing and/or implementing strategies to remove impediments to the
use of the ECI website
This is a report on the methods and results for Phase 1 and Phase2, and recommends strategies for Phase 3. It proceeds by describing prior research on factors affecting website usage then describes the phases in turn.
Page 9/31
2.4 Prior Research 2.4.1 ECI Review of Internet Access and Usage in Emergency Departments
Of particular relevance was the ECI’s 2014 Review of Internet Access and Usage in Emergency Departments1, where lack of suitable Internet access in NSW EDs was identified as being widespread.
The proportion of direct clinical staff in NSW with “poor or no Internet access” at work varied from 46% (Junior or Resident Medical Officer) to 75% (Enrolled Nurse) depending on respondents role. The proportion of staff across all ED roles with “poor or no Internet Access” varied by LHD as indicated in the following graph.
Figure 1 - Percentage of respondents by LHD reporting “poor” or “no” Internet access at work in ED
Lack of access to the Internet may obviously be a factor in poor use of the ECI website. Several specific barriers to Internet access were identified, as shown in the following graph, and it is relevant to the current study to identify if any of them were interfering with access to the ECI website in particular. 89% of ED staff surveyed stated that the following barriers “slightly” or “significantly” affect their work.
1 NSW Agency for Clinical Innovation, Review of Internet Access and Usage in Emergency Departments, January 2014, http://www.ecinsw.com.au/sites/default/files/field/file/ACI13-014-ed-internet-survey-web.pdf.
11%17%
28%29%
33%35%36%37%
43%47%
54%60%
64%68%
74%77%
80%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Mid North Coast*St Vincent's and Mater Health Network*
Hunter New EnglandWestern NSW*Northern NSW
Sydney Children's Hospital Network*Nepean Blue Mountains*
Northern SydneyWestern Sydney*
South Western SydneyCentral Coast*
Far West*Murrumbidgee*
SydneySouthern NSW
Illawarra ShoalhavenSouth Eastern Sydney
Percentage of respondents (* indicates sample of less than 30 respondents)
Page 10/31
Figure 2 - Percentage of respondents reporting the following barriers to Internet access at work
2.4.2 Broader Research on Website Usage
Technology Acceptance Model
A widely used and validated model on adoption of technology is the Technology Acceptance Model, which has been applied to various technologies including websites2. In its initial formulation, the Technology Acceptance Model identified two factors as determining a user’s intended and actual use of a technology:
Perceived usefulness: defined as “the degree to which a person believes that using a particular system would enhance his or her job performance”.
Perceived ease-of-use: “the degree to which a person believes that using a particular system would be free from effort”.
A later formulation identified a third influence:
The subjective norm: ‘‘a person’s perception that most people who are important to him think he should or should not perform the behaviour in question”.
The Technology Acceptance model has been criticised as being over-simplistic and incomplete, by ignoring other factors which determine the use of a technology, either directly or by influencing the perceived usefulness or ease of use3.
2 The Technology Acceptance Model was originally published as Davis F, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly, 13(3): 319–340 1989. Meta-analyses of the relevant literature are the following papers: King W, He J, A meta-analysis of the technology acceptance model, Information & Management 43 (2006) 740–755; Schepers J, Wetzels M, A meta-analysis of the technology acceptance model: Investigating subjective norm and moderation effects, Information & Management 44 (2007) 90–103. 3 Chuttur M, Overview of the Technology Acceptance Model: Origins, Developments and Future Directions ," Sprouts: Working Papers on Information Systems, 9(37) 2009. http://sprouts.aisnet.org/9-37
15%
35%
46%
48%
51%
51%
73%
0% 20% 40% 60% 80%
Other
Needing to log out of one system and log backin to another
Volume of other staff needing to use samecomputer
Poor/no wireless access for smartphones
Slow computer/outdated technology
Lack of computer/insufficient computers
Blocking firewalls/blocked sites
Percentage of respondents
Page 11/31
Specific Research on Websites
The most comprehensive and rigorous published analysis in this field is Wolk and Theysohn4 which examined websites with paid content. It identified only the following factors as statistically significant influences on website usage:
Accessibility: defined as the ease of locating the website by use of search engines and links from other sites.
Quality: the technical quality of the website, including such factors as speed of loading pages, quality of formatting, search capability within the website, and others.
Interactivity: the presence or otherwise of comments, discussion forums, chat, feedback, or similar mechanisms
Relevance: this was defined as the number of individual webpages in the whole web on the same topic of this website. The more webpages there are on the whole web on this topic, the more “relevant” it was deemed to be.
These results accord with the Technology Acceptance Model in the following way: “accessibility” and “quality” measures are likely to be determinants of “perceived ease-of-use,” and “relevance” is likely to be a proxy measure of other factors which influence “perceived usefulness”. “Interactivity” is probably a determinant of both “perceived ease-of-use,” and “perceived usefulness”.
Another study by Lederer et al. used a voluntary survey where respondents identified a single website useful to their work, and self-reported their frequency of website use on a scale from “frequently” to “infrequently”5. Respondents also answered numerous questions analysing their perceptions of characteristics of the website. There were statistically significant correlations with self-reported frequency of website usage with the following groups of propositions (none of the propositions had a significant correlation individually; statistical significance was achieved for when the results were aggregated according to the named groups: “ease of understanding” etc.):
“Ease of understanding o The site uses understandable graphics o The site uses consistent graphics o The site uses consistent terms o The site uses understandable terms o Display pages provide links to more detailed information o The site displays visually pleasing design o The display pages within the site are easy to read”
“Ease of finding o The site allows easy return to previous display pages o I can determine my position within the site o The site is easy to navigate”
4 Wolk A, Theysohn S, Factors influencing website traffic in the paid content market, Journal of Marketing Management, 2007, Vol. 23, No. 7-8, pp. 769-796. 5 Lederer, Albert L., et al. "The technology acceptance model and the World Wide Web." Decision support systems 29.3 (2000): 269-282.
Page 12/31
“Information quality o I use this site for accurate information for my job o I use this site for thorough information for my job o I use this site for timely information for my job o I use this site for relevant information for my job”
Lederer et al. explicitly linked their work to the Technology Acceptance Model: the first two groups were assumed to be determinants of “perceived ease-of-use”, and “information quality” was assumed to be a determinant of “perceived usefulness”.
Relevance to this project
The Technology Acceptance Model provides some guidance to this project. Heavy users of the website must perceive the website as being both useful and acceptably easy-to-use. One or other of those perceptions is likely absent in groups who are light users of the website. However, the Model itself provides no guidance on specific reasons why the perceptions may be present or absent.
The work by Wolk and Theysohn does suggest a small group of factors which may influence perceived ease-of-use or usefulness. However, the small group is unlikely to be comprehensive, and not all of them are necessarily relevant to the ECI website as opposed to their work on websites with paid content.
The analysis approach by Lederer et al. is relevant to understanding why heavy users of the ECI website are indeed heavy users, but is less useful for determining why light users are not. In particular, it assumes familiarity with the website which is likely to be absent amongst groups where most people do not use the site.
We used the Technology Acceptance Model and the other research listed above as a guideline for a questionnaire on why light users of the website are as such. However we use a more exploratory approach to identifying factors influencing perceived ease-of-use and usefulness, rather than assuming that they are as identified by Wolk and Theysohn, Lederer et al, or any other author. In addition the ECI’s earlier research has identified that difficulties with Internet access are likely to be a major influence on “ease of use”, and these are explicitly asked about.
Page 13/31
3. Phase 1: Analyse website usage
3.1 Aim This phase has identified LHDs which are heavy and light users of the ECI website.
3.2 Method 3.2.1 The ECI website uses Google Analytics to collect data on its use
The ECI website uses Google Analytics. A small piece of code is added to each webpage on the ECI site to allow Google Analytics to collect certain data about the users of the website6.
Phase 1 of the project involved:
1. Configuration of Google Analytics to collect some additional data about ECI website use
2. Collection of data over 6 weeks (9 September to 20 October 2014) 3. Analysis of the data
3.2.2 Google Analytics identifies the city or town of each website user
By default, Google Analytics collects numerous pieces of data about each session on the website. The data relevant to this project is:
Location of the user by country, state and town/city Which particular web page(s) each user visits
This default data set partly met the needs of this project. For example, it was reasonable to assume that most website visits from Newcastle were from the Hunter New England LHD. However, approximately 2/3 of website visits from NSW were from Sydney, and individual suburbs (or hospitals or LHDs) within Sydney could not be identified. Consequently, the default data from Google Analytics gathered insufficient information to identify the location, either physical or LHD, of about 2/3 of NSW website visitors.
3.2.3 eHealth NSW provided additional information to identify users by LHD
Each computer or device connected to the Internet uses a unique Internet Protocol address (IP address) to identify itself and allow data to reach it. Devices within any given organisation access the Internet from a range of IP addresses unique to that organisation.
On request, the then Information Services section of HealthShare NSW, now “eHealth NSW”, provided details of the IP address ranges used by each LHD in NSW. Normally, eHealth
6 The data collected by Google Analytics is insufficient to identify any individual person accessing the website, is covered by a privacy policy, and is stored in accordance with the ISO 27001 standard for information security. See support.google.com/analytics/topic/2919631 for further details. In addition, Google Analytics data relating to the ECI website is password-protected and accessible only to authorised people within the ECI.
Page 14/31
NSW would not centrally collate this information as each LHD manages its own Internet access. However, eHealth NSW had collated details of the IP addresses as part of a project several months before.
The IP address data had some limitations
It was several months out-of-date. In this time, several LHDs had changed their IP address ranges, e.g. by changing Internet Service Providers. It was not considered feasible in the timeframe of this project to contact each such LHD to obtain updated information.
With one exception (Royal Prince Alfred Hospital), it was possible to identify users only by LHD, not by individual hospital.
In many cases, two particular LHDs were still combined, for the purposes of internet access, along old Area Health Service boundaries.
The IP address data could identify only those users in each LHD who were using devices connected to the LHD network. It cannot identify the LHD if an individual was using a smartphone or tablet connected to a network other than an LHD Wi-Fi network.
3.2.4 Google Analytics was configured to identify use by each LHD
Google Analytics was configured to collate data about use of the ECI by each LHD, identified using the IP address information just discussed. Data collected during a pilot phase (9-21 September 2014) proved sufficient to:
Identify the LHD of approximately half of the NSW users of the ECI website Identify usage from 9 of the 15 LHDs in NSW
This coverage was considered sufficient for the purposes of the project. Data collection was extended to the period 9 September to 20 October 2014 prior to analysis.
3.2.5 The number of website sessions was compared to the number of ED presentations
The NSW Health Information Exchange automatically collects de-identified data on ED presentations from every hospital in NSW that uses an Electronic Medical Record in the ED, which includes all hospitals except a small number of rural ones. The Health Information Exchange was queried to obtain data on the total number of ED presentations for each LHD and hospital for the year 1/7/13 to 30/6/14. The number of ED presentations was used to normalise the results on website usage: it allowed website use to be expressed as the number of website sessions per 1,000 patient presentations.
The normalisation relied on an approximation: it was assumed that the ED presentations for every LHD during the study period (9/9/14 to 20/10/14) were at the average rate for that LHD the 2013-2014 year. It was considered unlikely that this approximation would introduce enough error to significantly affect the results.
Page 15/31
3.3 Results 3.3.1 Details & Discussion
Detailed results are in the following table:
Table 1 - ECI website use by LHD: Detail
For five LHDs, the IP address information supplied by eHealth NSW gave sensible data that could be additionally be verified by comparison to geographic information. These were:
Far West & Western NSW LHDs (combined data) Hunter New England Mid North Coast and Northern NSW
For example, website sessions identified by Google Analytics as being from Newcastle, Tamworth, or Armidale totalled 710. This number was, as expected, similar to but slightly larger than, the 635 sessions identified as being from the Hunter New England LHD’s IP address range. The excess sessions (710-635) represented access from home computers, computers at organisations outside the LHD, or mobile devices.
Data for
Illawarra Shoalhaven & South-Eastern Sydney (combined data) Royal Prince Alfred Hospital
were also collected according to the IP address information supplied by eHealth NSW. The data for Illawarra Shoalhaven and South Eastern Sydney could only partly be verified by reference to geographic information from Google Analytics, as it was not possible to identify
LHD(s)
Known
Website
Sessions by
IP
Assumed
Sessions By
Geography
(excludes
Sydney)
Assumed
Sessions Comments
ED
Presentations
2013‐2014
Pageviews/
1000
Presentations
Central Coast 136 136 Unreliable
Far West & Western NSW 165 179 165 Probably reliable 147441 9.73
Hunter New England 635 710 635 Probably reliable 392791 14.05
Illawarra Shoalhaven & South Eastern Sydney 849 334 849 Probably reliable 353709 20.86
Mid North Coast & Northern NSW 87 106 87 Probably reliable 295867 2.56
Murrumbidgee 236 236 Marginally reliable 137327 14.93
Nepean Blue Mountains & Western Sydney 4 No (good) data
Northern Sydney No (good) data
South Western Sydney & Sydney Excluding RPAH 1 No (good) data
Southern NSW 203 203 Marginally reliable 102972 17.13
Royal Prince Alfred Hospital 665 665 Probably reliable 73373 78.76
Sydney Childrens Hospitals 13 No (good) data
Start Date 9/09/2014
End Date 20/10/2014
Total Days 42
Total allocated views 2976
Total NSW views 6064
Page 16/31
which website access from Sydney was in the South Eastern Sydney area. The Royal Prince Alfred Hospital data could not be verified for the same reason. However, it was considered reliable as this hospital was in the unique position of having its own specific IP address range.
For eight LHDs, the IP address information supplied by eHealth NSW was found to be no longer correct, as no access occurred to the ECI website from the IP address ranges given. It was considered extremely unlikely that zero access would occur from any LHD, given the heavy access from others. For five of these LHDs, no good information could be obtained by geographic location, because no finer detail was available than “Sydney”. The LHDs were:
Nepean Blue Mountains Northern Sydney South Western Sydney Sydney (except Royal Prince Alfred Hospital) Western Sydney
For the remaining three LHDs, at least some information could be obtained from geographic location:
Central Coast Murrumbidgee Southern NSW
The Central Coast location data was considered an unreliable indication of access from the Central Coast LHD. Information from eHealth NSW suggested that at least some of the Central Coast LHD Internet access would be directed through the Northern Sydney LHD data centre, which in turn suggested that Google Analytics would not correctly identify the website access as being from the Central Coast as opposed to Sydney. No such concerns applied to the location data relating to the other two LHDs, which were considered marginally reliable. Results from the relevant locations (with most access from either Wagga Wagga or Goulburn, respectively) were broadly similar to the results for LHDs for which good data was available, which suggested they were valid. However, it is recognised that the resulting figures will be a moderate overestimate, as they will likely include some access from non-LHD sites.
3.3.2 Summary
Results considered sufficiently: reliable are as follows:
Figure 3 - ECI website use by LHD: Summary
2.56
9.73
14.05
14.93
17.13
20.86
78.76
0.00 20.00 40.00 60.00 80.00
Mid North Coast & Northern NSW
Far West & Western NSW
Hunter New England
Murrumbidgee
Southern NSW
Illawarra Shoalhaven & South Eastern…
Royal Prince Alfred Hospital
Website sessions per 1,000 patient presentations
Page 17/31
Note: IP address data has some limitations. It was only possible to reliably identify users by LHD, not by individual hospitals (with the exception of Royal Prince Alfred Hospital as indicated above).
The Royal Prince Alfred Hospital and, to a lesser extent Illawarra Shoalhaven & South Eastern Sydney LHDs stand out as heavy users of the ECI website. Conversely, the Mid North Coast and Northern NSW LHDs stand out as light users of the ECI website7.
4. Phase 2: Identify impediments to use of the website
4.1 Aim This phase collected qualitative data on impediments to use of the ECI website, by focussing on the LHDs which are the lowest users of the website, namely Mid North Coast LHD and Northern NSW LHD (“low-use LHDs”).
4.2 Method 4.2.1 Information was sought from ED staff
Information was sought from two groups of EDs in the low-use LHDs:
The largest EDs by annual patient presentations, as these collectively represent the majority of patient presentations, and hence the majority of potential use of the ECI website
A selection of medium-sized and smaller EDs, to elicit issues which may be unique to them
The largest EDs by patient presentation were selected, with an arbitrary cut-off of 20,000 presentations annually. Together these accounted for 62% of the presentations in the low-use LHDs in 2013-2014 financial year:
Coffs Harbour (36,196 annual presentations 2013-2014) Grafton (22,632) Kempsey (21,265) Lismore (29,675) Port Macquarie (29,829) Tweed (45,188)
7 It was considered possible that at least some of the website sessions from the Northern NSW LHD, in particular those at The Tweed Hospital, would be classified by Google Analytics having a location of “Gold Coast” in Queensland. There were 32 such sessions during the study period. Even in the unlikely event that all of these sessions were from The Tweed Hospital rather than the Gold Coast University Hospital or elsewhere in Queensland, the conclusions of this analysis would be the same. In particular, the Mid North Coast & Northern NSW LHDs would remain the lowest identified users of the ECI website.
Page 18/31
Smaller EDs arbitrarily selected were:
Ballina (16,432) Bellinger River (Bellingen) (4,087) Dorrigo Plateau (3,016) Maclean (11,369) Mullumbimby (7,949) Murwillumbah (17,127)
The names of the ED Director and ED Nurse Unit Manager (NUM) at each site were located from ECI records, and those people were contacted by email and phone. In several cases, other staff were contacted, either on recommendation from one of the above staff or because of difficulty contacting them. The other staff contacted included an Emergency Physician other than the ED Director, and senior nurses such as ED Clinical Nurse Consultants and a Nurse Practitioner. A response was obtained from at least one staff member at each site.
4.2.2 Questionnaire
A questionnaire was administered by phone to the contacted staff and results collated. Notes were taken on descriptive answers which were transcribed immediately afterward. The questionnaire is included in the appendix of this report. It includes a number of structured and open questions on factors influencing use of the ECI website.
4.3 Results 4.3.1 Responses
The major numerical results are summarised in Table 2. Results were generally consistent (note the generally tight interquartile ranges), and allowed several themes to be identified consistently across several responses. Relevant text responses are quoted later.
Table 2 - Responses: Main Items
The ECI website was found to be generally easy to access, and easy to use. The main deficit identified was in knowledge of the ECI’s website, with the majority of respondents either weakly agreeing or weakly disagreeing with the proposition that “the ECI’s website is well known in this department”. Respondents had variable agreement with the proposition “the ECI’s resources are useful in this department,” with a mean of weak agreement.
4.3.2 Factors encouraging use of the ECI website
As discussed above, knowledge of the ECI website was the weakest area identified in the survey. Amongst sites where the website was more well-known, the greatest influences identified amounted to ‘success breeding success’: resources on the ECI website were commonly used, and staff encouraged each other to use them.
Question Key Mean
How well do you know the ECI [1=not at all to 6=thoroughly] 4.86 [ 4.25 – 6 ]
The ECI’s website is well known in this department [1=strongly disagree to 6=strongly agree] 3.43 [ 3 – 4 ]
Our site finds the ECI website is easy to access [1=strongly disagree to 6=strongly agree] 5.29 [ 4.25 – 6 ]
The ECI website is easy to use [1=strongly disagree to 6=strongly agree] 5.15 [ 5 – 6 ]
The ECI’s resources are useful in this department [1=strongly disagree to 6=strongly agree] 4.14 [ 2.5 – 5.75 ]
Interquartile Range
Page 19/31
The role of leading by example in introducing other staff to resources such as the ECI website was mentioned in comments by several respondents (who are identified by site only to maintain anonymity):
"[With respect to all staff in ED,] I can show them a factsheet [on the ECI website] with a few clicks, and they say 'I never knew about that'" – Port Macquarie
“A visiting FACEM [emergency specialist] made staff aware of the website and knowledge spread like wildfire.” – Grafton
Figure 4 – Factors encouraging use of the ECI website
“People develop their own individual collections of resources. […] They often don't change until they see someone else using something.” – Lismore
And conversely, similar encouragement can impede use of the ECI website:
“[a staff specialist] from Tweed comes down and runs sim[ulation]s, shows us where to find things on the Tweed website” [with the implication that staff look at the Tweed website for resources rather than the ECI website] – Mullumbimby
4.3.3 Factors impeding use of the ECI website
Amongst sites where the ECI resources were less well known, the directed questions in the questionnaire proved less informative. Respondents indicated that there had been no teaching about the ECI website in their ED, but that was also true of departments where the ECI website was more commonly used - compare the items on “teaching” in Figures 4 and 5. The sites were generally using alternative sources for similar information, which is unsurprising. Specific alternative sources of information identified were:
Clinical Information Access Portal (CIAP) (3 sites) NSW Institute of Trauma and Injury Management (ITIM) website (1 site) UpToDate, a commercial website publishing guidelines and factsheets (1 site) NSW Health A-Z (2 sites) Resources on the local Intranet (3 sites)
More informative, however, were the comments by respondents. Several respondents identified a lack of knowledge of the ECI website amongst junior and locum doctors compared to permanent staff:
17%
29%
33%
100%
100%
0% 20% 40% 60% 80% 100%
Other
Steps have been taken to facilitate access
Formal encouragement , e.g. teaching
Site frequently uses specific pages
Informal encouragement
Percentage of respondents agreeing
Page 20/31
“We have a lot of new staff who are not as familiar with the website […] Junior staff tend to have a lot of things to find out [i.e. learning about the ECI website competes with other on-the-job learning]” - Kempsey
“We're staffed about half by regular staff and half by locums. The regular staff have good awareness of the website but the locum staff do not.” – Grafton
Figure 5 - Factors impeding knowledge of the ECI website
“Staff don't have time to get to know what resources are on the website.”- Coffs Harbour
Lack of awareness of the content of the ECI website was further illustrated by one respondent suggesting that we add clinical guidelines on a particular topic, when comprehensive guidelines on this topic had in fact been on the website for over a year.
One reason for staff not using the ECI website was that their existing pattern of use of use of online resources did not include it:
“Junior staff often prefer to use UpToDate, which is not necessarily ‘up to date; or reliable, and is often prepared by people at resident or registrar level. ECI is a preferable resource as it is NSW-based and peer-reviewed. […] “I would like the end-product of this process to be increased awareness of the ECI [website] amongst junior staff.” – Coffs Harbour
“I've never seen a doctor use the ECI website. We're staffed by GPs after hours and they use [their usual sources].” – Bellingen
“Everyone is aware of the ECI, but it is not one of their usual resources.” – Lismore
Mobile access was mentioned as a specific issue by two respondents:
“Junior staff usually have their own way of accessing information, often by apps. Does ECI have an app?” - Coffs Harbour
“Do you […] have an app? A lot of people just use their iPhones [to access clinical resources].” – Lismore
11%
67%
89%
89%
0% 20% 40% 60% 80% 100%
Website/Internet difficulties
Other
There has been no teaching about theECI website
Site uses alternative sources ofinformation
Percentage of respondents agreeing
Page 21/31
4.3.4 Internet access
No respondents reported a lack of access to the Internet by ED staff. However numerous sites, most of which were in the Northern NSW LHD, reported specific difficulties with Internet connections. Half of respondents described slow Internet access as being an issue for their ED(s)8. All except one of the respondents from the Northern NSW LHD agreed with the questionnaire item “The Internet access is too slow” whereas only one from the Mid North Coast LHD (at Dorrigo) did. Internet speed was identified as an issue at all three of the larger EDs in the Northern NSW LHD. Specific comments were:
"With our small sites, it's taken a long time to get them to use the Intranet." "Sometimes the Internet just doesn't work at all” – Tweed and surrounding EDs
“The speed of the Internet [as opposed to Intranet] is very variable. Staff seek resources on the Intranet as they load faster and more reliably. It would be better if the ECI website was available by Intranet.” – Grafton
Access to video content on the ECI website9 was specifically blocked from some or all of the same sites:
“Any link to a social media site - Facebook, YouTube and so on is blocked. […] videos on the ECI website don't work.” – Lismore
"We don't have access to YouTube." – Tweed Hospital & surrounding EDs
Difficulties with both Internet access in general, and video in particular, were being addressed by the local IT services. It appears that video may have been blocked because of slow Internet access rather than security policy:
"Some sites have got [their own Internet] gateways [and video now works at those sites]" – Tweed and surrounding EDs
8 Respondents generally made a distinction between “access” to the ECI website and the speed of the Internet. The statement ”Our site finds the ECI website is easy to access” received strongly affirmative results (see Table 3), whereas many respondents then went on to indicate that Internet speeds were sometimes slow. Results in this survey were somewhat in contrast to of the ECI’s 2014 Review of Internet Access and Usage in Emergency Departments (see section 1.3.1). In the earlier survey, 33% of respondents in the Northern NSW LHD, and 11% of respondents in the Mid North Coast LHD, reported “poor or no” Internet access. A difference in questions partly explains the contrast in the results: the earlier survey did not distinguish ability to access the Internet versus speed of the Internet, whereas the current survey did. In addition, the earlier study more extensively surveyed more junior medical and nursing staff, who may have experienced somewhat less access to the Internet than then respondents in the current study. 9 The ECI actually hosts its own video content on Vimeo. The site is mis-identified as YouTube by the respondents, presumably because they were not able to see the video site at all.
Page 22/31
4.3.5 Usability of the ECI Website
Usability of the website was rated highly by respondents.
Table 3 - Responses to usability questions
The Item with lowest agreement was “It is easy to navigate and find resources on the ECI website”. The responses to this item had a bimodal distribution, with most respondents finding the website easy to navigate, but a minority having difficulties.
“Navigation of the website can be difficult. For example, the distinction between ‘Clinical’ and ‘Clinical Support’ headings is unclear” – Coffs Harbour
A substantial minority of respondents had difficulty accessing content linked from the ECI website. This was generally video content, as discussed in the previous section. One exception was:
“Any bad experiences of the ECI website, e.g. the head injury advice links being broken, puts off staff from using the site again.” – Coffs Harbour10
In contrast, no respondent described difficulty with reading and display of material hosted on the ECI website itself.
4.3.6 The usefulness of the ECI website
As discussed earlier, respondents had variable agreement with the proposition “the ECI’s resources are useful in this department,” with a mean of weak agreement.
79% of respondents identified the patient factsheets as being as “particularly useful or commonly used”. One specific comment was:
“We're only a small ED so we're only looking for the basic stuff [with respect to patient factsheets” [i.e. respondent was satisfied with those that were there] – Murwillumbah
The patient factsheets were the only resource identified as “particularly useful or commonly used” by more than one site. Others mentioned by individual respondents were:
The Oxylog guide (regarding a commonly-used ventilator) Clinical pathways, e.g. thromboembolism Nurse Delegated Emergency Care Nurse Management Guidelines Models of care
10 On investigation, a specific problem was found with a link from the ECI website to a head injury factsheet hosted on the ITIM website. The link was partly broken, in that it redirected to the ITIM homepage rather than the relevant factsheet.
Question Key Mean
The ECI website is easy to use [1=strongly disagree to 6=strongly agree] 5.15 [ 5 – 6 ]
It is easy to navigate to find resources on the ECI Website [1=strongly disagree to 6=strongly agree] 4.92 [ 4.75 – 6 ]
The search capabilities on the ECI Website are helpful… [1=strongly disagree to 6=strongly agree] 5.00 [ 5 – 6 ]
The graphic design of the ECI website is pleasant and unobtrusive [1=strongly disagree to 6=strongly agree] 5.46 [ 5 – 6 ]
The pages on the ECI website are easy to read [1=strongly disagree to 6=strongly agree] 5.92 [ 6 – 6 ]
The pages on the ECI website display correctly without errors [1=strongly disagree to 6=strongly agree] 5.77 [ 6 – 6 ]
Interquartile Range
Page 23/31
Figure 6 - Answers to "what would make the ECI website more useful"
In answer to “what would make the ECI website more useful”, 86% of respondents nominated “better awareness of the ECI website”, which was the only response affirmed by the majority of respondents.
Two respondents suggested emails from the ECI as a means of raising awareness of the ECI, e.g. regarding new content:
“Also, email or similar prompts would be helpful.” - Coffs Harbour
"Maybe just an email out to everyone just to remind them.” – Tweed and surrounding EDs
When prompted, a few respondents nominated particular new resources they would find useful:
“We previously used the ICCMU [NSW Intensive Care Coordination and Monitoring Unit] website for drug protocols. It would be useful if similar information was available on the ECI website.”- Grafton11
“Suitable review articles [on relevant medical conditions]” - Coffs Harbour
“The Oxylog guide is very good - more along those lines would be good.” – Kempsey
“If you can sell it as a one-stop shop then it becomes easier for everyone to default onto it. [It is important that] for each thing you know where to go.” – Grafton
4.4 Analysis and Summary of Findings 4.4.1 Awareness of the ECI website is fundamental
There is good basic awareness of the ECI itself, but that does not generally extend to frequent use of the ECI’s resources in the target LHDs. Where the ECI website is most widely known, it is known by virtue of its use: certain ECI resources are frequently used, and the ECI website become visible to staff because they see it used. The role of what can be termed a ‘local champion‘ was identified by several respondents. Conversely, individuals’ habitual pattern of access to information was identified by several respondents as an
11 The ICCMU website was found to no longer host drug protocols.
7%
29%
29%
86%
0% 20% 40% 60% 80% 100%
Other
Better resources on the ECI website
Better access to the Internet or the ECIwebsite
Better awareness of the ECI website
Percentage of respondents agreeing
Page 24/31
impediment to use of the ECI website. There was near-universal agreement that better awareness of the ECI website would make it more useful.
The ECI is primarily seen as a source of patient factsheets. There is substantially less awareness of clinical tools and other resources. This may, in turn be affecting perceptions of usefulness of the ECI website.
Several respondents identified a particular lack of awareness of the ECI website amongst junior and locum staff compared to permanent staff.
4.4.2 Poor Internet access impedes access to the ECI website in the Northern NSW LHD
In the Northern NSW LHD specifically, the speed and reliability of Internet access was identified as specific issue for access to the ECI website: access to the Internet is slower and sometimes less reliable than access to intranet sites, and specific resources linked from the ECI website, notably videos, could not be accessed.
4.4.3 Usability of the ECI website is generally good
Usability of the website was rated highly by respondents, with only a minority having difficulties navigating the website.
A technical deficit was discovered with respect to broken links on the ECI website: although the ECI website is regularly checked for broken links, the current mechanism does not detect the sort of error discussed on page 22, where a broken link redirects to a valid page such as the home page of the site being linked to. This particular error was identified by one respondent as undermining confidence in the ECI website.
4.4.4 Results from this phase are consistent with the Phase 1 results
The above factors provide a sufficient explanation of the Phase 1 results identifying that the Mid North Coast and Northern NSW LHDs used the ECI website substantially less than other LHDs. Two factors were identified which are likely to disproportionately apply to the Mid North Coast and Northern NSW LHDs:
Poor Internet access in the Northern NSW LHD makes the ECI website a less attractive source than Intranet sites. Other LHDs are less likely to have similar difficulties.
Unawareness of the ECI website was identified particularly amongst rotating junior and locum staff. It is likely that locum staff, in particular, are proportionally more prevalent in the Mid North Coast and Northern NSW LHDs than in the metropolitan LHDs.
The remainder of factors are likely to apply more widely across NSW.
Page 25/31
5. Phase 3: Recommended actions
5.1 Raising awareness of the ECI website 5.1.1 Raising awareness amongst permanent staff
While the ECI’s website resources are not generally well known in the target LHDs, the ECI itself is fairly well known, at least amongst permanent staff. Consequently, the overarching goal amongst permanent ED staff is to convert a general awareness of the ECI to specific awareness and use of ECI resources.
Actions which should be considered are:
Recommendation 1. The ECI send out regular, e.g. quarterly, emails on ‘what is new on the ECI website’, which would serve both to introduce new content and as a reminder of what is there.
Recommendation 2. Resources on the ECI website be demonstrated as a matter of routine at ECI symposia and forums. Again, this would serve both to introduce new content and as a reminder of what is there. Such demonstrations may be more effective if conducted by ED staff who use the resources rather than ECI staff. Any surrounding discussion would also be helpful.
5.1.2 Raising awareness amongst junior and locum staff
Amongst junior and locum staff, knowledge of ECI resources has been identified as lower than amongst permanent staff, and knowledge of the ECI itself is likely to be lower. These groups present particular difficulties with respect to communications from the ECI:
Rotating prevocational junior medical officers (JMOs) present a particular difficulty for communications from the ECI, as they spend only a minority of their time in ED, which implies that any ED-specific information from the ECI will be relevant to them only a minority of the time.
Locum staff obviously tend to be itinerant and are disconnected from many channels of communication within LHDs, and from the ECI to ED staff.
Access to email mailing lists of relevant locums and junior staff will vary from indirect (e.g. contacting junior doctors via the NSW Health Education and Training Institute--HETI) to non-existent.
Junior staff and new long-term staff are overwhelmed by information during orientation sessions, and the effectiveness of adding more information during them will be poor. Indeed the suggestion of such is likely to be resisted by staff conducing orientations.
Within EDs, education about the ECI resources is largely informal: staff show other staff what is available. Raising awareness about ECI amongst the above groups of staff is a complex task involving multiple facets:
Raising awareness of both the ECI itself and the ECI resources Facilitating and stimulating informal education about the ECI resources Encouraging, where applicable, formal education about the ECI resources
Page 26/31
Although several possible methods for obtaining these goals were considered, none carry an unambiguous likelihood of success at a reasonable cost. Consequently all of the following are introduced as possibilities rather than definite recommendations. An alternative to them all is to instead concentrate on raising awareness of the ECI website amongst permanent ED staff, and rely on existing processes of informal or formal teaching of other staff.
Recommendation 3. (Possible) The ECI provide minor computer equipment of practical use to EDs, such as mousepads or mice, printed with ECI branding and messages, to act as a passive reminder of the ECI at the point of information access, with a goal of raising awareness of the ECI, and stimulating informal discussion about the ECI.
Recommendation 4. (Possible) The ECI produce a one-page flyer, in both printed and electronic versions, introducing the ECI website and specifically what resources are available on it. Consequently, the ECI would then identify and contact the staff member(s) at each ED responsible for producing orientation packs for locums, rotating junior staff, and/or new permanent or longer-term staff, encouraging :
Inclusion of a flyer (just discussed) Identifying any particular ECI resources frequently used at the ED
Full implementation would require regular reminders, particularly preceding the start of Junior Medical Officer terms.
Such an action is more likely to be successful with respect to locums than for the other groups, as the latter tend to be more overwhelmed with new information. In addition this recommendation would represent a substantial amount of work by the ECI of uncertain benefit.
Recommendation 5. (Possible) The ECI produce a slide-set or video introducing the ECI website and specifically what resources are available on it, aimed at use during education sessions in EDs. Consequently, the ECI would then identify and contact relevant staff at those EDs that conduct regular staff teaching sessions, providing the above material
Again this recommendation would represent a substantial amount of work by the ECI. However, provision of teaching material is likely to be viewed positively by those responsible for regular teaching.
Recommendation 6. (Possible) The ECI identify material on the ECI website which is useful to prevocational JMOs on wards outside the ED. This may include much of the ECI’s material on procedures, and many of the clinical tools. Occasional or periodic emails would then be sent to prevocational JMOs informing them of these resources. This would require access to a mailing list or lists of JMOs, probably through HETI. Such an action would raise awareness of the ECI resources amongst JMOs irrespective of their current role. It may, however, be controversial as it may be seen as outside the remit of the ECI.
5.2 Internet access difficulties The Northern NSW LHD’s Internet access difficulties are most likely unique to that LHD. Information from one respondent suggests that Internet access is progressively being improved on a site-by-site basis. Rectification of the difficulties is a local issue for the LHD, and the ECI has little scope to assist it.
Page 27/31
Recommendation 7. (Possible) The ECI contact the Chief Information officer of the Northern NSW LHD to provide feedback on the Internet access difficulties reported by respondents there.
5.3 ECI website usability 5.3.1 General usability
As mentioned above, usability of the current website was rated highly by respondents, with only a minority having difficulties navigating the website. The ECI website is due to have its hosting arrangements and design changed within the next few months, which may affect usability. Depending on the degree of change, the following may be considered:
Recommendation 8. (Possible) Following the ECI website redesign, opinions could be sought again on usability, either by a questionnaire or by seeking comments by some other means, e.g. a discussion at a symposium or forum.
5.3.2 Broken links
Recommendation 9. As part of the ECI website redesign, the mechanism for detecting broken links should be reviewed, to ensure that it detects redirected links as well as completely broken ones.
Page 28/31
6. Summary & Conclusions The goal of this project is to increase the effectiveness of the ECI website in providing resources of use to ED clinicians in NSW.
6.1 Phase 1 Phase 1 successfully identified LHDs which were high and low users of the ECI website, and in particular identified the Mid North Coast and Northern NSW LHDs standing out as light users of the ECI website. This allowed Phase 2 of the project to proceed to identify factors impeding use of the ECI website.
6.2 Phase 2 Phase 2 involved a survey to elect details of factors impeding use of the website in the LHDs using it least. It was successful in identifying such factors consistently across multiple responses.
Most of the identified factors impeding use of the ECI website were likely to be generally present across multiple LHDs:
Awareness of the ECI website is fundamental
o There is generally good awareness of the ECI itself, but that does not generally extend to frequent use of the ECI’s resources. Where the ECI website is most widely known, it is known by virtue of its use: certain ECI resources are frequently used, and the ECI website becomes visible to staff because they see it used. There was agreement that better awareness of the ECI website would make it more useful
o The ECI is primarily seen as a source for patient factsheets. There is substantially less awareness of clinical tools and other resources
o There was a particular lack of awareness of the ECI website amongst rotating junior and locum staff compared to permanent staff.
Usability of the ECI website is generally good
o Usability of the website was rated highly by respondents, with only a minority having difficulties navigating the website.
o A specific technical deficit was discovered with respect to broken links on the ECI website.
Some factors were likely to be disproportionately applicable to the LHDs surveyed, namely:
There were specific difficulties with Internet access in one of the LHDs As mentioned earlier, unawareness of the ECI website was identified amongst locum
staff. It is likely that locum staff, are proportionally more prevalent in the LHDs surveyed than in the metropolitan LHDs.
Page 29/31
6.3 Phase 3 The above factors were analysed and recommendations made on actions to promote use of the ECI website. Raising awareness of the ECI amongst locum and more junior staff is particularly challenging, and possible actions are all of uncertain benefit.
At the time of writing, the ECI is considering the recommended actions and will complete Phase 3 of the project by implementing suitable ones.
Page 30/31
7. Appendix: Phase 2 Questionaire
7.1 Description & Purpose The following is the questionnaire administered by phone during Phase 2 of the project, in order to elicit qualitative information regarding impediments to usage of the ECI website.
7.2 Questions General questions
1. I have consulted other staff members before answering this survey [Yes/No] a. If so, what job categrory(s)
2. How well do you know the ECI [Likert scale: thoroughly well to not at all]?
How well do you agree with the following statements:
3. The ECI’s website is well known in this department [Likert scale] a. If agreeing: what factors contribute to the knowledge of the ECI website in this
department [all that apply]: Formal encouragement, e.g. demonstrations at teaching sessions Informal encouragement, i.e. people find them useful and encourage
others to use The department frequently uses one or more specific pages on the
website Which one(s)
Steps have been taken to facilitate access e.g. intranet links Other, please specify
b. If disagreeing: to your knowledge, what factors contribute to the lack of knowledge of the ECI website in this department [all that apply]:
There has been no teaching (formal or informal) about the ECI website within the department
The ECI website (or Internet in general) is difficult or impossible to access
We have alternative sources of similar resources which are suitable for our needs
Which source(s) Other, please specify
4. Our site finds the ECI website is easy to access. (Take into account all relevant
issues, including access to computers, access to the Internet, and access to the ECI website itself.) [Likert scale/Unknown]:
a. If affirmative: which of the following steps have been taken to facilitate access to the ECI website:
Enabling unrestricted access to the Internet Requesting IT support to unblock the ECI website
Page 31/31
Providing intranet links Other, please specify
b. If disagreeing: what problems with access to the ECI website do you have [all that apply]:
The Internet is blocked or unavailable from all computers at this site The Internet is blocked or unavailable from most computers at this site The Internet is available but the ECI site is blocked Access to the Internet requires logins and passwords that are not
commonly available Access to the Internet requires logging out and logging in again, or a
similar time-consuming process There are too few computers in suitable areas at this location to be
able to use the Internet when we need it The computers available are too slow or outdated The Internet access is too slow Other, please specify
5. The ECI website is easy to use (Likert scale/Unknown)
Unless unknown, Likert scales for: It is easy to navigate to find resources on the ECI Website The search capabilities on the ECI Website are helpful and easy to use The graphic design of the ECI website is pleasant and unobtrusive The pages on the ECI website are easy to read The pages on the ECI website display correctly without errors
If negative, what browser and version?
6. The ECI’s resources are useful in this department (Likert scale) a. If affirmative: are any resources particularly useful or commonly used? If so,
please specify. b. All answers: what would make the ECI resources more useful:
Better awareness of the ECI website Better access to the Internet or the ECI website Better resources on the ECI website
Better in what way? More coverage of useful topics
Give examples of what your ED would find useful More comprehensive coverage of topics
Give examples if possible More up-to-date and/or accurate coverage of topics
Give examples if possible Other, please specify
7. Do you have any other comments?
