A randomized controlled trial comparingin-person and wiki-inspired nominal grouptechniques for engaging stakeholders inchronic kidney disease researchprioritizationMeghan J. Elliott1,2* , Sharon E. Straus1,3, Neesh Pannu4, Sofia B. Ahmed5, Andreas Laupacis1,2,3,George C. Chong6, David R. Hillier7, Kate T. Huffman8, Andrew C. Lei9, Berlene V. Villanueva10, Donna M. Young11,Helen Tam-Tham12, Maoliosa Donald5, Erin Lillie1, Braden J. Manns5,12 and Brenda R. Hemmelgarn5,12
Abstract
Background: Few studies have evaluated stakeholder engagement in chronic kidney disease (CKD) researchprioritization. In this two-arm, parallel group randomized controlled trial, we sought to compare an in-personnominal group technique (NGT) approach with an online wiki-inspired alternative to determining the top 10CKD research priorities, and to evaluate stakeholder engagement and satisfaction with each process.
Methods: Eligible participants included adults ≥18 years with access to a computer and Internet, high healthliteracy, and from one of the following stakeholder groups: patients with CKD not on dialysis, their caregivers,health care providers who care for patients with CKD, or CKD-related health policymakers. Fifty-six participantswere randomized to a wiki-inspired modified NGT that occurred over 3 weeks vs. a 1-day in-person NGT workshop,informed by James Lind Alliance methodology, to determine the top 10 CKD-related research priorities. The primaryoutcome was the pairwise agreement between the two groups’ final top 10 ranked priorities, evaluated usingSpearman’s correlation coefficient. Secondary outcomes included participant engagement and satisfaction andwiki tool usability.
Results: Spearman’s rho for correlation between the two lists was 0.139 (95 % confidence interval −0.543 to 0.703,p = 0.71), suggesting low correlation between the top 10 lists across the two groups. Both groups ranked the sameitem as the top research priority, with 5 of the top 10 priorities ranked by the wiki group within the top 10 forthe in-person group. In comparison to the in-person group, participants from the wiki group were less likely toreport: satisfaction with the format (73.7 vs.100 %, p = 0.011); ability to express their views (57.9 vs 96.0 %, p = 0.0003);and perception that they contributed meaningfully to the process (68.4 vs 84.0 %, p = 0.004).(Continued on next page)
* Correspondence: [email protected] Ka Shing Knowledge Institute, St. Michael’s Hospital, 209 Victoria Street,East Building, Toronto, Ontario M5B 1T8, Canada2Institute of Health Policy, Management and Evaluation, University ofToronto, 155 College Street, Toronto, Ontario M5T 3M6, CanadaFull list of author information is available at the end of the article
Conclusions: A CKD research prioritization approach using an online wiki-like tool identified low correlation in rankingscompared with an in-person approach, with less satisfaction and perceptions of active engagement. Modifications tothe wiki-inspired tool are required before it can be considered a potential alternative to an in-person workshopfor engaging patients in determining research priorities.
Trial registration: (ISRCTN18248625)
Keywords: Chronic kidney disease, Research priorities, Patient preferences, Wiki
Abbreviations: CI, Confidence interval; CKD, Chronic kidney disease; CONSORT, Consolidated Standards of ReportingTrials; eGFR, Estimated glomerular filtration rate; JLA, James Lind Alliance; NGT, Nominal group technique;PCORI, Patient-Centered Outcomes Research Institute; SPOR, Strategy for patient oriented research
BackgroundChronic kidney disease (CKD), defined as estimatedglomerular filtration rate (eGFR) <60 mL/min/1.73 m2,affects approximately 5–7 % of the adult population [1–3]and contributes to excess morbidity, mortality and healthcare costs [4, 5]. Despite the tremendous burden CKDplaces on health care delivery, care for persons affected byCKD remains suboptimal [6]. These care gaps may resultfrom the lack of research that is relevant to end-users, in-cluding patients with CKD, their caregivers, and the clini-cians involved in their care [7, 8]. To address these gaps,there is increasing emphasis on the importance of en-gaging key stakeholders in determining research prioritiesand establishing user-relevant research objectives [9, 10].The Strategy for Patient Oriented Research (SPOR) [11] inCanada and the Patient-Centered Outcomes ResearchInstitute (PCORI) in the United States [12] are two initia-tives that aim to engage patients in research and knowledgetranslation and to focus on interventions and outcomesthat are important to research end users.Although few established methods for involving key
stakeholders in research prioritization exist, the JamesLind Alliance (JLA) priority setting partnership model iswidely considered an inclusive and transparent processand has been applied to a number of disease conditions[13]. The final step of the four-step JLA partnership in-volves an in-person workshop with a nominal grouptechnique (NGT) to use a list of the top 30 research pri-orities to develop a ranked list of the top 10 researchpriorities from key stakeholders through small and largegroup activities [14]. While considered the referencestandard, this approach has some clear limitations forparticipants and coordinators, including time and re-source requirements, travel, and logistical consideration[15]. These challenges may be particularly problematicfor patients with chronic illness who have difficulty withor are unable to travel, which could limit participant in-volvement and generalizability of findings. AlternativeInternet-based approaches to research priority setting,such as an interactive wiki-like platform, could obviate
some of these limitations. A ‘wiki’ is a collaborative web-based platform that allows users to add and edit contentonline in real time, and in the context of researchprioritization can be used to summarize the preferencesof multiple stakeholders [16]. Wiki-inspired tools havebeen successfully applied in similar, multi-stakeholderconsensus approaches, including the creation of anasthma action plan [16], but have not been studied inresearch prioritization exercises that require significantpatient involvement.In this randomized controlled trial we sought to com-
pare the in-person workshop and NGT approach withan online wiki-inspired alternative as the final step in theJLA research prioritization process for key CKD stake-holders, including patients with CKD not on dialysis, theircaregivers, and the health care professionals and policy-makers involved in their care. We aimed to compare thetop 10 research priorities resulting from each process andto evaluate the engagement and satisfaction of partici-pating stakeholders with their assigned intervention.
MethodsThis trial is reported in accordance with the ConsolidatedStandards of Reporting Trials (CONSORT) 2010 statement[17] (trial registration: www.isrctn.org, ISRCTN18248625).
Overview of the JLA CKD priority setting partnershipThe JLA priority setting partnership uses establishedmethods to identify and prioritize unanswered disease- ortreatment-related questions [14]. Our CKD priority settingprocess involved: 1) creation of a 12-person steering groupof patients with non-dialysis CKD, caregivers, cliniciansand researchers; 2) identification of research uncertaintiesthrough a national survey of representative CKD stake-holders; 3) refinement and prioritization of the researchuncertainties identified in the survey to determine a short-list of the top 30 research priorities by the steering group;and, 4) participation by stakeholders from across Canadain a 1-day priority setting workshop to determine thetop 10 CKD research priorities from a shortlist of 30.
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 2 of 12
Steps 1–3 took place from July 2014 to June 2015; thistrial involved step 4 only.
Trial participantsThe steering group enlisted the support of nephrologynetworks, partner organizations, and nephrologists fromacross Canada to identify potential participants. Eligibleparticipants were ≥18 years of age, English speaking, hadaccess to and were comfortable using high-speed Internet,and had high health literacy (determined by asking the fol-lowing question: “How confident are you in filling outmedical forms by yourself?” [18]). Participants were alsorequired to be a member of a stakeholder group: patientswith CKD not on dialysis or with a prior transplant;informal caregivers of persons with CKD (relatives,family members, or friends who help patients managetheir illness); health care professionals (primary carephysicians, nephrologists, nurses, pharmacists, socialworkers, or dietitians) who care for patients with CKD;or health policymakers (non-clinicians with the abilityto influence or determine policies and practices relatedto health care delivery for CKD at an international,national, regional, or local level). The following indi-viduals were not eligible to participate: patients receivingdialysis and/or who had received a kidney transplant; per-sons with an underlying diagnosis of dementia or cogni-tive impairment; and patients deemed unfit to travel. Aresearch assistant confirmed the eligibility of potentialparticipants, described the trial to them in detail, andobtained their written informed consent.
InterventionsParticipants were randomized 1:1 to an in-person NGT-based workshop or an online wiki-inspired platform.Randomization occurred by random number sequencegeneration using a central electronic system to ensureallocation concealment. Patients and their caregiverswere randomized as a dyad to avoid contamination.Additionally, the 6 patient and caregiver steering groupmembers were allocated to the in-person workshopgroup only, and not randomized, as they had originallycommitted to participate in the NGT process of thestudy and it was therefore deemed important to includethem in the NGT arm. Participants in both groups wereprovided with the top 30 uncertainties shortlist (derivedfrom a national survey and prioritized by the steeringgroup) to review and consider prior to the intervention;participants in the in-person workshop group rankedthe 30 uncertainties prior to the workshop, while par-ticipants in the wiki-inspired group ranked the top 10(due to logistical issues of ranking 30 online).Individuals in the 1-day in-person workshop group
participated in a NGT approach [14, 19] in Toronto,Canada, which included a combination of small and
large group exercises facilitated by four individuals withexperience in the JLA process. The final ranking of thetop 10 research priorities was determined by large groupconsensus, with voting when needed.Wiki group participants were provided with a 1-h
training webinar on use of the wiki and were givenunique usernames and passwords to access the securewebsite before they began the following tasks to deter-mine the top 10 research priorities. The wiki-basedintervention took place online over 3 phases: 1) anindividual ranking phase (7 days), where participantssubmitted their preferences for the top 10 researchpriorities individually, 2) a group ranking phase (12 days),where participants determined the final top 10 researchpriorities collaboratively, and 3) a sign-off phase (2 days),where participants could review to confirm their agree-ment, but not re-order the final list. Following theindividual phase, individual rankings were aggregatedacross participants using reverse scoring, such thatitems ranked first received a score of 10, and thoseranked last received a score of 1. These scores wereaggregated and presented as a summary rank list towhich participants could refer throughout the collabo-rative ranking phase. During the collaborative rankingphase, participants could re-order the top 10 prioritiesin real time and discuss their selections using a chatfeature. The chat feature consisted of a text box anddisplay of all participant comments in chronologicalorder, organized by unique user identifier (ie identifyinginformation and stakeholder role were not apparentduring chat to avoid differential perception of input byprovider role). A facilitator monitored the site daily andstimulated discussion through comments using the chatfeature. Technical support was available throughout theprocess. Due to the nature of the interventions, studyparticipants and intervention administrators were notblinded to group assignment; however, statistical ana-lysis was blinded. The in-person workshop took placeduring the wiki process; participants in both groupswere blinded to the ranking results of the other group.
OutcomesThe primary outcome was the pairwise agreement betweenthe two groups’ final top 10 ranked priorities recordedupon conclusion of the workshop and group ranking phasefor the in-person and wiki groups, respectively. Secondaryoutcomes included perceived engagement of participantsin their assigned prioritization process and participantsatisfaction. We also evaluated usability aspects of theonline wiki tool including content, features, and acces-sibility (see Additional file 1 for details on outcomesand methods of measurement). The electronic question-naire assessing participant engagement and satisfactionand wiki usability was administered using the FluidSurveys
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 3 of 12
platform (FluidSurveys platform, Fluidware Inc., Ottawa,Ontario)., Key elements of stakeholder engagement wereevaluated in the participant engagement section of thequestionnaire, including trust, respect, accountability,legitimacy, fairness, and competence [20, 21].
Sample sizeThe primary outcome was the pair wise agreement betweenthe two groups’ top 10 priorities, therefore the correlationanalysis was driven by the number of research prioritiesconsidered, which was fixed at 10 for the purposes of thisstudy. As the JLA recommends 20 to 30 participants inthe in-person priority setting exercise, with approximatelyequal representation of patients, clinicians, and caregivers,we estimated that a total of 40–60 participants would berequired (we aimed for 50). Inflating this by 10 % to ac-count for potential dropouts, an overall; sample size of 56was felt to be a conservative estimate for the primaryoutcome.
Statistical analysesBaseline demographic information was compared betweenthe in-person workshop and wiki groups using univariateanalyses. The unit of analysis for the primary outcomewas at the group level, and the pairwise agreementbetween the two groups’ priorities (provided as ranks)was examined using Spearman’s correlation coefficientand its associated 95 % confidence interval (CI). Wealso compared the rank order of the top 10 researchpriorities from each group descriptively, noting the pro-portion that were within the top 10 across both groups.Our secondary outcomes were obtained by survey forboth the in-person and wiki groups, with responses mea-sured using a 5-point Likert scale (where 1 = stronglydisagree, 5 = strongly agree), and the proportions ofparticipants who agreed with each statement (4 or 5 onthe scale) were compared between groups using chi-square tests. Written comments also obtained from thesurvey by both groups were examined qualitatively. Tworesearchers (MJE and BRH) reviewed all written com-ments and independently coded them for emergingcategories and themes. Resulting themes and discrepan-cies were discussed, and a final coding scheme describingparticipants’ experiences with each process was estab-lished. Participant observation data collected duringthe in-person workshop was summarized quantitatively(ie frequencies) and qualitatively (ie brief researcher notes).The time requirements to undertake each process werecompared descriptively.
ResultsParticipantsFifty-six individuals were randomized to either the in-person workshop (n = 28) or online wiki-inspired group
(n = 28). In addition the 6 patient and caregiver steeringgroup members were allocated to the in-person work-shop group only, as established a priori, for a total of 34participants for the in-person workshop. Overall 25 indi-viduals took part in the 1-day workshop (12 patients, 6caregivers, 6 health care professionals, and 1 policy-maker), and 26 individuals participated in the 3-weekonline wiki-like process (12 patients, 8 caregivers, 5health care professionals, and 1 policymaker). Participantenrolment and reasons for withdrawal are outlined inFig. 1. During the 3-week period of wiki activity, 21 of26 participants (80.8 %) logged onto the ranking page atleast once during either the individual or group phase(3 patients and 2 caregivers did not access the site duringthe intervention). All participants contributed to the pri-mary analysis of agreement between the two top 10 re-search priority lists, as this was assessed at the group level.Only participants who completed the post-interventionsurvey were included in the analysis of participant engage-ment and satisfaction.The majority of study participants were 40–64 years of
age, and 47 % were male (Table 1). 12 patients and 6caregivers participated in the in-person workshop, and11 patients and 8 caregivers participated in the onlinewiki-inspired process. Compared to the in-person group,a smaller proportion of patients and caregivers in thewiki-inspired group were < 40 years of age and female(5.6 vs. 21.1 % and 38.9 vs 52.6 % respectively), and theywere less likely to work full time (31.6 vs. 44.4 %) orhold a university degree (26.3 vs. 61.1 %).
Comparison of the top 10 CKD research prioritiesAt the conclusion of the in-person workshop and wiki-inpsired interventions, the top 10 research priorities weredetermined for each group (Table 2). Overall, Spearman’srho for correlation between the two groups was 0.139(95 % CI −0.543 to 0.703, p = 0.71), suggesting a non-significant low level of correlation between the twogroups. Both groups independently ranked the sameitem as the top research priority (What are the mosteffective new interventions and treatments to preventthe development and progression of kidney disease?). Ofthe top 10 research priorities ranked by the wiki-inspiredgroup, 5 were ranked within the top 10 by the workshopgroup, and a further 4 items were ranked within the top15 by the workshop group. Of note, 5 of the top 10 re-search priorities in the NGT group were not in the top 10of the wiki-inspired group, with four of these being healthservices research questions. The #10 priority in the NGTworkshop group was extensively discussed in both smalland large group settings, suggesting that greater inter-action in the NGT workshop setting may have influencedthe final rankings. The research priorities that were com-mon to the top 10 lists of both groups included dietary
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 4 of 12
measures to slow CKD progression, CKD symptoms, andthe impact of lifestyle factors and medications on kidneydisease. Two wiki group members (both clinicians) hadparticipated previously in an in-person priority settingworkshop for patients on or nearing dialysis [22], whichmay have influenced their perceptions and experiences ofthe intervention.
Participant satisfaction and engagementAll 25 participants in the NGT-based workshop groupand 19 of 26 participants from the online wiki-inspiredgroup completed the post-intervention questionnaire.The non-responders from the wiki group included 4 pa-tients, 2 caregivers and 1 health care professional. Allworkshop participants felt the in-person NGT formatwas well-suited (ie noted agreement or strong agree-ment) to determine CKD research uncertainties, whereas73.7 % of respondents in the wiki group were satisfied(agree/strongly agree) with the format (p = 0.011) (Fig. 2a).Ninty-six percent of participants in the workshop groupfelt they could adequately express their views in this for-mat, as compared to 57.9 % in the wiki group (p = 0.003).With respect to participant engagement, workshopparticipants were significantly more likely than wiki
participants to report that the format used by thegroup respected the ideas of all participants (100 vs.68.4 %, p = 0.004) and perceived that their views con-tributed to the final list of research priorities (84.0 vs.33.3 %, p < 0.001) (Fig. 2b). Respondents from bothgroups felt they had adequate knowledge (80.0 % inworkshop group vs. 83.3 % in wiki group, p = 1.0) andskills (92.0 % in workshop group vs. 89.5 % in wikigroup, p = 1.0) to rank the top research priorities, andaround one quarter of participants felt that influentialparticipants impacted the group dynamics (28.0 % inworkshop vs. 21.1 % in wiki group p = 0.73).Written comments were provided in the survey by 11
participants in the in-person NGT group and 10 partici-pants in the wiki-inspired group. The main themes thatemerged were related to opportunities for personal inter-actions and meaningful contributions, participationrates, format effectiveness and the importance of com-munication to justify individual rankings (Table 3), withthe workshop group generally providing more positivefeedback of their experience across these three themes.Written comments from wiki participants highlightedtheir perceptions of a low level of active participationand inadequacy of the chat feature (which aimed to
Fig. 1 Flow diagram outlining intervention group allocation and withdrawals. *Primary outcome measured at the group level
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 5 of 12
allow comprehensive discussion of participants’ views,preferences and justification of changes to the theirrankings). Wiki participants felt that the top ranked un-certainties were ordered and re-ordered without a clearunderstanding of the reasons for the changes and withlimited group discussion. In contrast, in-person work-shop participants felt they were able to contributeequally and effectively among all stakeholders in thisformat.Observation of participants during the in-person work-
shop revealed a moderate to high degree of verbal andphysical engagement for the majority of group mem-bers. Three participants (2 patients and 1 caregiver)
demonstrated limited verbal and physical interactionwith others. Most individuals appeared to readily voicetheir views, questions and comments spontaneously orwith prompting by the facilitator. While the initial smallgroup discussions were reserved and initiated mainly bythe facilitator, more spontaneous and animated discus-sions among the participants arose throughout the day.Some participants contributed more confidently and per-suasively than others, but these individuals were notlimited to one stakeholder group and did not appear tonegatively dominate any of the group discussions. In thewiki-inspired group, chat discussions were similarlyreserved and required facilitator prompting initially.
Table 1 Baseline demographics of participants in the in-person workshop and online wiki groups, n (%)
In-person workshopgroup (n = 25)
Patients and caregivers inin-person workshop (n = 18)
Online wiki-inspiredgroup (n = 26)
Patients and caregivers in onlinewiki-inspired group (n = 19)
Age
18–39 years 3 (12) 1 (5.6) 4 (15.4) 4 (21.1)
40–64 years 14 (56) 9 (50) 16 (61.5) 9 (47.4)
≥ 65 years 8 (32) 8 (44.4) 6 (23.1) 6 (31.6)
Sex
Male 13 (52) 11 (61.1) 11 (42.3) 9 (47.4)
Female 12 (48) 7 (38.9) 15 (57.7) 10 (52.6)
Role
Patient 12 (48) 12 (66.7) 11 (42.3) 11 (57.9)
Caregiver 6 (24) 6 (33.3) 8 (30.8) 8 (42.1)
Health Care Provider 6 (24) - 6 (23.1) -
Policymaker 1 (4) - 1 (3.8) -
Marital Status
Married 22 (88) 16 (88.9) 19 (73.1) 13 (68.4)
Common Law 1 (4) 0 (0) 2 (7.7) 1 (5.3)
Divorced/Separated 2 (8) 2 (11.1) 0 (0) 0 (0)
Single 0 (0) 0 (0) 3 (11.5) 3 (15.8)
Not Indicated 0 (0) 0 (0) 2 (7.7) 2 (10.5)
Employment Status
Full-time 14 (56) 8 (44.4) 12 (46.2) 6 (31.6)
Part-time or casual 2 (8) 1 (5.6) 2 (7.7) 1 (5.3)
Retired 9 (36) 9 (50) 10 (38.5) 10 (52.6)
Unemployed 0 (0) 0 (0) 1 (3.8) 1 (5.3)
Not Indicated 0 (0) 0 (0) 1 (3.8) 1 (5.3)
Highest Level of Education
Some High School 0 (0) 0 (0) 1 (3.8) 1 (5.3)
High School Graduate 4 (16) 4 (22.2) 3 (11.5) 3 (15.8)
College Diploma 4 (16) 3 (16.7) 8 (30.8) 8 (42.1)
University Degree 17 (68) 11 (61.1) 12 (46.2) 5 (26.3)
Not Indicated 0 (0) 0 (0) 2 (7.7) 2 (10.5)
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 6 of 12
However, throughout the process, participants providedspontaneous input and sought each other’s views in arespectful manner.
Wiki usabilityDuring the 3-week period of wiki activity, 21 of 26 par-ticipants (80.8 %) logged onto the ranking page at least
Table 2 Top 10 CKD-related research priorities resulting from the in-person workshop vs. online wiki-based groups
In-person workshop group Wiki rank Online wiki-based group Workshop rank
1. What are the most effective newinterventions and treatments to preventthe development and progression of kidneydisease?
1 1. What are the most effective newinterventions and treatments to prevent thedevelopment and progression of kidneydisease?
1
2. What is the best diet to slow progressionof kidney disease and what are the benefitsand risks of specific diets (ie phosphaterestriction, protein restriction, low salt etc.)in terms of kidney disease progression andquality of life?
6 2. What are the harmful effects ofmedications used in patients with CKD,and in particular, the combinations ofmedications used to treat other diseases(such as diabetes and high blood pressure)?
7
3. What are the causes of symptoms inpatients with chronic kidney disease,including fatigue, low energy, sleepingproblems, depression, anxiety and sexualdysfunction, and how can these best betreated to improve quality of life?a
8 3. What are the best signs or markers(ie blood tests, urine tests or other tests) toidentify and diagnose kidney disease early?
14
4. What are the optimal strategies, suchas having access to health information(eg lab test results), sharing of information,and/or improving communication, to helppatients manage their health condition(s)themselves and to improve patientexperience and outcomes?
Not ranked in the top 10 4. What are the benefits and risks associatedwith use of vitamins, supplements andalternative/complementary therapies(ie herbal, naturopathic, marijuana etc.) interms of kidney disease progression andquality of life?
11
5. What is the impact of lifestyle factors(ie exercise, stress) on risk of developingkidney disease, kidney disease progression,and quality of life?
9 5. How can we predict how fast kidneyfunction will get worse, and when kidneyswill fail?
13
6. What are the optimal strategies for themanagement of CKD (ie those undertakenby the primary care physician, nephrologist,other health care professionals) to delayprogression and improve outcomes?
Not ranked in the top 10 6. What is the best diet to slow progressionof kidney disease and what are the benefitsand risks of specific diets (ie phosphaterestriction, protein restriction, low salt etc.)in terms of kidney disease progression andquality of life?
2
7. What are the harmful effects ofmedications used in patients with CKD,and in particular the combinations ofmedications used to treat other diseases(such as diabetes and high blood pressure)?
2 7. What are the optimal medications(eg ACE inhibitors, ARBs, phosphate binders,sodium bicarbonate, etc.) to slowprogression of kidney disease?
19
8. What are the optimal approaches for theprevention and treatment of cardiovasculardisease in patients with CKD?
Not ranked in the top 10 8. What are the causes of symptoms inpatients with chronic kidney disease,including fatigue, low energy, sleepingproblems, depression, anxiety and sexualdysfunction, and how can these best betreated to improve quality of life?a
3
9. What is the best strategy (eg screening,programs targeting high risk groups,programs to increase public awareness)to identify kidney disease early?
Not ranked in the top 10 9. What is the impact of lifestyle factors(ie exercise, stress) on risk of developingkidney disease, kidney disease progression,and quality of life?
5
10. How do we ensure that patientswith CKD have equitable access to care(eg nephrologists, allied health clinics)irrespective of location of residence orsocio-economic status?
Not ranked in the top 10 10. How can communication regardingpatient care be improved and/orstreamlined across all disciplines(primary care, nephrology, allied health)to improve outcomes and the patientexperience?
12
CKD chronic kidney disease, ACE angiotensin-converting enzyme, ARB angiotensin II receptor blockersaItem subsumed a second uncertainty noted within the top 30 uncertainties in both the in-person and wiki groups
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 7 of 12
once during either the individual or group phase. Thosewho did not log in during the intervention included 3 pa-tients and 2 caregivers. During the group ranking phase,13 users (50.0 %) were considered active participants(ie they made direct changes to the top 10 uncertaintieslist [n = 11, 42.3 %] and/or participated in the chat feature[n = 13, 50.0 %]). These active participants included 6 pa-tients, 3 caregivers, and 4 health professionals. One patientwho was admitted to hospital during the individual rankingphase accessed the ranking tool, submitted her individualranking, and participated in the group ranking and discus-sion from a laptop computer while in hospital.Over the 12-day group ranking phase, a total of 192
changes were made to the rankings (mean 16 changes/day, range 1–37) and 104 comments were provided inthe group chat (mean 8.7 comments/day, range 2–21).The most changes were made on the first day of the groupround (37 changes). Twenty-six changes were made to thelist on the final day, and the most group discussion(21 comments) occurred on the final day of the exer-cise. The most active participants included 2 patientsand 1 health care provider, each of whom provided between46 and 58 contributions to the rankings and/or group chat.Half of the wiki-inspired group participants viewed andconfirmed they had reviewed the final top 10 list followingconclusion of the group ranking phase.Results from the wiki usability portion of the post-
intervention questionnaire are outlined in Additional file 2.
Of the 18 respondents, all felt that 12 days were sufficientto undertake the group ranking process, and 61 % used thetool as often as they would have liked. Eighty-three percentof participants experienced no difficulties in changing therankings, and 61 % found the chat feature useful for dis-cussing the uncertainties as a group. Six participants con-tacted study coordinators regarding technical issues, all ofwhich related to difficulty logging onto the site. All issuesbut one were resolved, with this user being able to loginafter repeated attempts but having ongoing difficulties par-ticipating in the ranking process.
Time requirementsThe in-person workshop intervention took place over 1weekday and required travel time for participants fromacross Canada to attend. The online wiki-based groupcould access the ranking site at any time of day, fromany location, and as often as participants preferred duringa 3-week period.
DiscussionIn this randomized trial evaluating two methods ofstakeholder engagement to determine CKD researchpriorities, we found that an online wiki-inspired ap-proach had a low level of correlation compared to theresearch priorities determined from an in-person NGTapproach informed by JLA methodology, with only 5 ofthe top 10 priorities identified by the wiki-inspired group
Fig. 2 a Satisfaction of workshop and wiki participants on post-intervention questionnaire. *p < 0.05 for between-group difference. b Engagementof workshop and wiki participants on post-intervention questionnaire. *p < 0.05 for between-group difference
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 8 of 12
within the top 10 priorities of the in-person group. Therewere also important differences between the groups, withfewer health-services research questions ranked by thewiki-inspired group, and differences in perceptions of en-gagement. In comparison to participants in the in-personworkshop, those in the wiki-inspired group were less satis-fied with the format of the exercise and felt they were lesslikely to contribute meaningfully to the process. Whereasall individuals who attended the in-person workshop con-tributed to the group discussions (either spontaneously orwith prompting), 80 % of individuals in the wiki groupaccessed the tool at least once, and only half were con-sidered ‘active’ participants. We are unable to deter-mine if the differences in research priorities identifiedwere due to the different processes, or differences in
the groups themselves (including the patient and caregiversteering group members participating in the in-personworkshop), as priorities may vary by group irrespective ofthe process used to establish them. This will be importantto address in future work.Our study is one of the first to evaluate an online wiki-
based tool to engage patients and other stakeholders inhealth care research prioritization and to undertake a ran-domized comparison of two distinct research prioritizationmethods. Although both techniques are based on a NGTapproach, the online wiki-inspired option had attractivefeatures including flexibility to contribute at a convenienttime during a process that occurred over a longer period oftime compared to the 1-day in-person workshop, whichthe majority of participants felt was adequate. However the
Table 3 Main themes identified from written feedback on post-intervention questionnaire
Group Role Comment
Limited Personal Interactions
Wiki Caregiver “It was easier not to put your whole heart into this as it would have been if it were done face toface with the other participants… I felt something was missing, some sort of connection withothers facing the same difficulties.”
Wiki Health Care Professional “The wiki seemed to reward the last person to change the rankings. In person, a doc such as mecould have had a more subtle way to influence the process without being aggressive or tryingto take it over.”
Importance of Communication to Justify Selections
Wiki Patient “When people explained their rationale for their preferences, I understood. When the rankingwas changed without using the chat feature to augment our understanding, I could not discernthe rationale for the preferences.”
Wiki Patient “Although I enjoyed the discussions that did occur and the mix of group members (caregiver,patient, clinician), I'm not sure, as a whole, we really made the most of the chat feature.”
Wiki Health Care Professional “I think any changes in the top 10 should have had comments to support these changes.”
Wiki Health Care Professional “In a discussion forum online, I also didn't feel comfortable asking specific people to clarifyor explain their choices.”
Format Effectiveness
Workshop Caregiver “The format enabled patients and caregivers to engage with a wide group of professionals,other than the traditional doctor-patient relationship.”
Wiki Patient “Basically, I felt this process was less effective and that we didn't actually reach a groupconsensus with our outcomes.”
Workshop Patient “The only part that was not entirely satisfactory was the final phase in which the entire groupworked to re-order priorities. I felt that it lacked the fluidity and finesse to allow for minorchanges.”
Perspectives Dependent on Participation
Wiki Patient “I was disappointed by the participation rate (those who did not participate). I think this affectedthe quality of the discussions and possibly weakened the final outcome.”
Workshop Health Care Professional “Some opinions/voices might not be represented depending on who attends (is able to attend)and who is approached to attend the workshop.”
Ability to Contribute Meaningfully
Workshop Caregiver “As a layperson I felt that even though I did not have the hard earned scientific expertise of themedical community, all our experiences were considered on their merits.”
Workshop Patient “The workshop was an amazing experience where a collaborative effort of all affected people byCKD joined together to come up with the top 10 important and highly timely researchuncertainties that will give better life with the patients.”
Workshop Patient “I left feeling that in some small way I had contributed to research possibilities for my diseasewhich currently has no cure.”
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 9 of 12
longer duration required for participation may be per-ceived as a disadvantage by some. The standard JLAmethodology for stakeholder involvement in health careresearch prioritization, including a final in-person work-shop, has been implemented to date in more than 20chronic diseases, including prior work by our group toidentify research priorities for patients on or nearing dialy-sis [22]. A recent systematic review on research prioritysetting in kidney disease identified 16 studies, only 4 ofwhich involved patients in the prioritization process [23].The studies cited in this review used a variety of methodsto elicit research priorities, none of which were describedin detail nor used an electronic or online collaborativeplatform to engage stakeholders, and none of which wereevaluated in a randomized trial such as this one.Wiki-like tools have been applied in other collabora-
tive health care-related activities involving patients andother stakeholders. Gupta et al. included patients andclinicians in a wiki-inspired collaborative design of anasthma action plan (WikiBuild), where groups of 14 par-ticipants contributed to the action plan’s content and de-sign over a 1-week period [16]. Other studies examiningthe application of wiki-based tools in patient-centeredresearch, including the development of clinical practiceguidelines [24] and patient information leaflets [25], havefound these tools to be promising alternatives to moretraditional approaches. In contrast to our study, theseother studies generally involved input from a large num-bers of patients (>300) and lack the personalized andmulti-stakeholder interactions endorsed by the JLA format.Potential advantages of an online wiki-inspired approach
for collaborative research prioritization include flexibilityin participation, lack of travel requirements limiting par-ticipant eligibility, and lower costs [16]. A particular ad-vantage is the ability to include those whose illnesses maypreclude easy travel; for example, one patient in the wiki-inspired group in our study was admitted to hospitalduring the study period and participated in the processwhile in hospital. The complex group dynamics often seenwith in-person techniques can lead to the development ofstatus hierarchies that favour clinicians over patients andcaregivers [19, 26, 27], which could limit the openness ofindividual contributions to the discussion and influencethe final product [28], although our qualitative resultswould suggest that was not the case in our study.Although neither the wiki group nor the in-person work-shop group indicated the presence of overly influentialindividuals, wiki-inspired participants indicated the abilityto contribute honestly and explicitly in this format.Although 61 % of wiki-inspired participants responding
to the post-intervention questionnaire felt that the chatfeature was useful for explaining their views and prefe-rences, many expressed concerns that this format did notallow for as comprehensive a discussion and understanding
as is possible with face-to-face interactions. Gupta et al. de-scribed similar participant concerns for the wiki onlinechat, as well as the lack of individual accountability forchanges made [16]. Other barriers to the adoption ofwiki-inspired tools for patient engagement activities de-scribed include concerns regarding suitability of the for-mat to its objectives, poor participation rates, and technicalchallenges, among others [16, 24, 25]. It is unclear whetherpatients with poor health literacy or cognitive impairmentwould be able to participate in a wiki-inspired priority set-ting process, and this is an area for future work. The par-ticipant feedback provided during our trial highlights theneed to refine the online collaborative tool, and the chatfeature in particular, to ensure participants feel engagedand able to contribute meaningfully to the process.Our study is one of the first to describe the application
of a wiki-inspired tool to a research prioritization exer-cise, with a randomized comparison of methods foreliciting input from a diverse group of stakeholders todetermine research priorities. The results, however, shouldbe interpreted in light of the study limitations. The re-quirement for participants to have regular access to acomputer and the Internet may have limited eligibleparticipants and thus the generalizability of findings.However, Internet access is increasingly common amongpatients with chronic disease (up to 70 % of patients withchronic illness are Internet users) [29], and approximatelyhalf of patients with advanced CKD access the Internet toobtain health information [30, 31]. Further, little evidenceexists to guide the evaluation of stakeholder engagementwith such consensus-driven processes [32]. While quanti-tative and written responses on the post-intervention sur-vey provide valuable information, additional insights couldbe gained through participant interviews evaluating indi-vidual experiences with the interventions. Finally, 6 of thepatient and caregiver steering group members participatedin the NGT in-person workshop, and were not part of therandomization process. Although these steering groupmembers were involved in the earlier stages of the CKDpriority setting partnership (ie identification of researchpriorities through national survey and refinement of thesepriorities into a top 30 shortlist), the final ranking of thetop 10 priorities was a collaborative process involving allstakeholders participating in the workshop. All partici-pants were encouraged to contribute to the final rankingprocess, and the top 10 priorities reflect group consensusrather than the views of any one individual.
ConclusionsInvolving patients and other stakeholders in determiningresearch priorities has the potential to enhance the qualityand relevance of research for end users. In our randomizedcontrolled trial of patients, caregivers, clinicians andpolicymakers, we found that an online collaborative
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 10 of 12
wiki-inspired process identified different top ten CKD-related research priorities compared with an in-personworkshop, and participants in the wiki group were lesslikely to feel engaged and satisfied. We are unable todetermine if the differences in research priorities iden-tified were due to the different processes or differencesin the groups themselves, as priorities may vary by groupirrespective of the process used to establish them. This isimportant to address in future work. Modifications to thewiki tool, including optimization of a communication fea-ture to improve the ability of participants to communicatetheir views and preferences, need to be designed andevaluated before a wiki process can be considered afeasible alternative to an in-person workshop for en-gaging patients in determining research priorities.
Additional files
Additional file 1: Table summarizing methods used for assessingprimary and secondary outcomes. (DOCX 104 kb)
Additional file 2: Figure demonstrating usability responses from thewiki-based group on the post-intervention questionnaire. (DOCX 120 kb)
AcknowledgementsWe would like to thank Mr. David Newton for his assistance in the developmentand administration of the online collaborative ranking tool. We also thankMs. Sarah Gil for her regular communication with and technical assistance to allparticipants in the wiki group, and Ms. Sarah Gillis for assisting with thecoordination of the in-person workshop.
FundingMJE is supported by an Alberta Innovates-Health Solutions Clinician Fellowshipand Kidney Research Scientist Core Education and National Training ProgramPost-Doctoral Fellowship. BRH, NP, SBA and BJM are supported by an alternativefunding plan from the Government of Alberta and the Universities of Albertaand Calgary. SBA is supported by an Alberta Innovates – Health SolutionsClinical Investigator award. HT is supported by an Alberta Innovates – HealthSolutions Graduate Studentship. AL is supported by a Canada Research Chair inHealth Policy and Citizen Engagement, SES is supported by a Tier 1 CanadaResearch Chair in Knowledge Translation, and BRH is supported by the Roy andVi Baay Chair in Kidney Research. This research was supported by the CanadianInstitutes of Health Research and an interdisciplinary team grant from AlbertaInnovates – Health Solutions, the Interdisciplinary Chronic Disease Collaboration.The sponsors of the study had no role in study design, data collection, dataanalysis, data interpretation or writing of the report.
Availability of data and materialThe data supporting the main conclusions of this article are included withinTable 2 (top 10 CKD-related research priorities resulting from the in-personworkshop vs. online wiki-based groups). All other data is stored in a securedata holding at the University of Calgary and may be made available fromthe corresponding author on reasonable request.
Authors’ contributionsMJE, BRH, and SES were responsible for the conception and design of thisproject. MJE, NP, SBA, AL, GCC, DRH, KTH, ACL, BVV, DMY and BRH weremembers of the JLA steering committee who contributed substantially to thedesign and implementation of the workshop and wiki interventions. HT, MD,EL, and BJM were additional non-steering committee individuals who providedmethodological expertise and significant contributions in the planning andadministration of the in-person workshop. MJE conducted and is responsiblefor the data analysis. All authors drafted or revised the article critically forimportant intellectual content, and all authors gave final approval of the versionto be published.
Competing interestsThe authors declare that they have no competing interests.
Consent for publicationNot applicable.
Ethics approval and consent to participateEthics approval for this study was obtained from the Institutional ReviewBoard of the University of Calgary. Written informed consent was obtainedfrom all participants.
Author details1Li Ka Shing Knowledge Institute, St. Michael’s Hospital, 209 Victoria Street,East Building, Toronto, Ontario M5B 1T8, Canada. 2Institute of Health Policy,Management and Evaluation, University of Toronto, 155 College Street,Toronto, Ontario M5T 3M6, Canada. 3Department of Medicine, University ofToronto, Suite RFE 3-805, 200 Elizabeth Street, Toronto, Ontario M5G 2C4,Canada. 4Department of Medicine, University of Alberta, 8440 112 Street,Edmonton, Alberta T6G 2G3, Canada. 5Department of Medicine, University ofCalgary, 1403 29 Street NW, Calgary, Alberta T2N 2T9, Canada. 6Suite 200,5970 Centre Street SE, Calgary, Alberta T2H 0C1, Canada. 7343 Webb Court,Newmarket, Ontario L3Y 5E8, Canada. 823-1140 Falcon Drive, Coquitlam,British Columbia V3E 2J6, Canada. 94K Spadina Avenue, Suite 1820, Toronto,Ontario M5V 3Y9, Canada. 10A4123-409 Tache Avenue, Winnipeg, ManitobaR2H 2A6, Canada. 1131 Seligs Road, Prospect Village, Nova Scotia B3T 2A6,Canada. 12Department of Community Health Sciences, University of Calgary,3280 Hospital Drive NW, Calgary, Alberta T2N 4Z6, Canada.
Received: 26 February 2016 Accepted: 18 August 2016
References1. Coresh J, Selvin E, Stevens LA, Manzi J, Kusek JW, Eggers P, et al. Prevalence
of chronic kidney disease in the United States. JAMA. 2007;298(17):2038–47.2. Arora P, Vasa P, Brenner D, Iglar K, McFarlane P, Morrison H, et al. Prevalence
estimates of chronic kidney disease in Canada: results of a nationallyrepresentative survey. CMAJ. 2013;185(9):E417–23.
3. Aitken GR, Roderick PJ, Fraser S, Mindell JS, O'Donoghue D, Day J, et al.Change in prevalence of chronic kidney disease in England over time:comparison of nationally representative cross-sectional surveys from 2003to 2010. BMJ Open. 2014;4(9), e005480.
4. Go AS, Chertow GM, Fan D, McCulloch CE, Hsu CY. Chronic kidney diseaseand the risks of death, cardiovascular events, and hospitalization. N Engl JMed. 2004;351(13):1296–305.
5. Jha V, Garcia-Garcia G, Iseki K, Li Z, Naicker S, Plattner B, et al. Chronic kidneydisease: global dimension and perspectives. Lancet. 2013;382(9888):260–72.
6. Ronksley PE, Hemmelgarn BR. Optimizing care for patients with CKD.Am J Kidney Dis. 2012;60(1):133–8.
7. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translationof research findings. Implement Sci. 2012;7:50.
8. Liberati A. Need to realign patient-oriented and commercial and academicresearch. Lancet. 2011;378(9805):1777–8.
9. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al.Mapping the impact of patient and public involvement on health andsocial care research: a systematic review. Health Expect. 2014;17(5):637–50.
10. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM,et al. How to increase value and reduce waste when research priorities areset. Lancet. 2014;383(9912):156–65.
11. Canadian Institutes of Health Research. Strategy for Patient-Oriented Research(SPOR) http://www.cihr-irsc.gc.ca/e/41204.html. Accessed 15 Aug 2015.
12. Patient-Centered Outcomes Research Institute (PCORI) http://www.pcori.org/.Accessed 15 Aug 2015.
13. Cowan K. The James Lind alliance: tackling treatment uncertainties together.J Ambul Care Manage. 2010;33(3):241–8.
14. Cowan K. The James Lind Alliance Guidebook. Oxford: James Lind Alliance; 2013.15. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in
systematic reviews: knowledge transfer for policy and practice. J ContinEduc Health Prof. 2008;28(2):67–72.
16. Gupta S, Wan FT, Newton D, Bhattacharyya OK, Chignell MH, Straus SE.WikiBuild: a new online collaboration process for multistakeholder tooldevelopment and consensus building. J Med Internet Res. 2011;13(4), e108.
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 11 of 12
17. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: Updatedguidelines for reporting parallel group randomised trials. Ann Intern Med.2010;152(11):726–32.
18. Powers BJ, Trinh JV, Bosworth HB. Can this patient read and understandwritten health information? JAMA. 2010;304(1):76–84.
19. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J,et al. Consensus development methods, and their use in clinical guidelinedevelopment. Health Technol Assess. 1998;2(3):i–iv. 1–88.
20. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, et al.Stakeholder participation in comparative effectiveness research: defining aframework for effective engagement. J Comp Eff Res. 2012;1(2):181–94.
21. Lavallee DC, Williams CJ, Tambor ES, Deverka PA. Stakeholder engagementin comparative effectiveness research: how will we measure success?J Comp Eff Res. 2012;1(5):397–407.
22. Manns B, Hemmelgarn B, Lillie E, Dip SC, Cyr A, Gladish M, et al. Settingresearch priorities for patients on or nearing dialysis. Clin J Am Soc Nephrol.2014;9(10):1813–21.
23. Tong A, Chando S, Crowe S, Manns B, Winkelmayer WC, Hemmelgarn B,et al. Research priority setting in kidney disease: a systematic review.Am J Kidney Dis. 2015;65(5):674–83.
24. den Breejen EM, Nelen WL, Knijnenburg JM, Burgers JS, Hermens RP, Kremer JA.Feasibility of a wiki as a participatory tool for patients in clinical guidelinedevelopment. J Med Internet Res. 2012;14(5), e138.
25. van de Belt TH, Faber MJ, Knijnenburg JM, van Duijnhoven NT, Nelen WL,Kremer JA. Wikis to facilitate patient participation in developing informationleaflets: first experiences. Inform Health Soc Care. 2014;39(2):124–39.
26. Pagliari C, Grimshaw J. Impact of group structure and process onmultidisciplinary evidence-based guideline development: an observationalstudy. J Eval Clin Pract. 2002;8(2):145–53.
27. Cantrill JA, Sibbald B, Buetow S. The Delphi and nominal group techniquesin health services research. Int J Pharm Pract. 1996;4:67–74.
28. Mansell I, Bennett G, Northway R, Mead D, Moseley L. The learning curve:the advantages and disadvantages in the use of focus groups as a methodof data collection. Nurse Res. 2004;11(4):79–88.
29. Kruse RL, Koopman RJ, Wakefield BJ, Wakefield DS, Keplinger LE, CanfieldSM, et al. Internet use by primary care patients: where is the digital divide?Fam Med. 2012;44(5):342–7.
30. Schatell D, Wise M, Klicko K, Becker BN. In-center hemodialysis patients’ useof the internet in the United States: a national survey. Am J Kidney Dis.2006;48(2):285–91.
31. Ong SW, Jassal SV, Porter E, Logan AG, Miller JA. Using an electronic self-management tool to support patients with chronic kidney disease (CKD):a CKD clinic self-care model. Semin Dial. 2013;26(2):195–202.
32. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement inresearch: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45.
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal
• We provide round the clock customer support
• Convenient online submission
• Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research
Submit your manuscript atwww.biomedcentral.com/submit
Submit your next manuscript to BioMed Central and we will help you at every step:
Elliott et al. BMC Medical Informatics and Decision Making (2016) 16:113 Page 12 of 12
