Choosing and using methodological search filters:searchers’ viewsSophie Beale*, Steven Duffy†, Julie Glanville‡, Carol Lefebvre§, Dianne Wright‡, Rachael McCool‡,
Danielle Varley‡, Charles Boachie¶, Cynthia Fraser¶, Jenny Harbour** & Lynne Smith***Liverpool Reviews and Implementation Group, University of Liverpool, Liverpool, UK, †Centre for Reviews and
Dissemination, University of York, York, UK, ‡York Health Economics Consortium, University of York, York, UK, §Lefebvre
Associates Ltd, Oxford, UK, ¶Health Services Research Unit, University of Aberdeen, Aberdeen, UK, and **Healthcare
Improvement Scotland, Glasgow, UK
Abstract
Background: Search filters or hedges are search strategies developed to assist information specialists andlibrarians to retrieve different types of evidence from bibliographic databases. The objectives of this pro-ject were to learn about searchers’ filter use, how searchers choose search filters and what informationthey would like to receive to inform their choices.Methods: Interviews with information specialists working in, or for, the National Institute for Health andCare Excellence (NICE) were conducted. An online questionnaire survey was also conducted and adver-tised via a range of email lists.Results: Sixteen interviews were undertaken and 90 completed questionnaires were received. The use ofsearch filters tends to be linked to reducing a large amount of literature, introducing focus and assisting withsearches that are based on a single study type. Respondents use numerous ways to identify search filtersand can find choosing between different filters problematic because of knowledge gaps and lack of time.Conclusions: Search filters are used mainly for reducing large result sets (introducing focus) and assistingwith searches focused on a single study type. Features that would help with choosing filters include mak-ing information about filters less technical, offering ratings and providing more detail about filter valida-tion strategies and filter provenance.
Keywords: database searching; databases, bibliographic; information services; librarians; methodologicalfilters; surveys
Key Messages
• Information specialists surveyed develop new search strategies at least once a week.• Nearly all information specialists surveyed use methodological search filters.• The majority of information specialists surveyed use different filters depending on whether their
search requires sensitivity or precision.• Half of information specialists surveyed sometimes amend published search filters.• Information on the sensitivity and precision of a search filter is an important factor for selection of
an appropriate filter.
Introduction
Search filters, also known as ‘hedges’ or optimalsearch strategies, have been developed to assist
information specialists and librarians, researchers,clinicians and other searchers to retrieve differenttypes of evidence from bibliographic databases.They consist of combinations of search terms, gen-erally drawing on both free text and controlledvocabulary (such as Medical Subject Headings in
Correspondence: Julie Glanville, York Health Economics Consortium(YHEC), University of York, York, YO10 5NH, UK, E-mail: [email protected]
Medline).1 They are typically combined with setsof subject terms.2–5
Search filters are an important tool for thosewho are searching for specific research designs orsubjects because they can save time in identifyingsome of the elements of the strategy and, wheresearch filters have been validated, can providetools whose reliability can be referenced.3,4 Searchfilters are a tool of particular relevance to searchersand researchers producing systematic reviews andclinical guidelines, because they can provide somestandardisation in a field where standards andguidance are sparse and sensitive retrieval isimportant.In 2004, a telephone and fax survey to deter-
mine the awareness and use of methodologicalsearch filters among health and scientific librariansin the United Kingdom reported a high level ofawareness but a low level of usage.2 Since 2004many more filters have been published.3,4 Filtersare now incorporated into database interfaces, suchas the PubMed Clinical Queries,4 as well as beingreported in medical and scientific journals.3
Research methods to develop search filtersinclude the identification of gold standards toderive and test strategies, the use of test and vali-dation sets of records to test the performance offilters, the use of statistical techniques to identifysearch terms and to test various search filters toidentify those which perform best in terms of sen-sitivity and precision.2 The methodology of searchfilter development is evolving.6 The use ofresearch methods to develop filters is improvingthe probability that search filters will be reliableand consistent in their performance when usedwith a range of subject strategies.5,7–9
As search filters have increased in number, criti-cal appraisal tools have been developed which canbe used to assess or appraise search filters.2,10,11
These critical appraisal tools can help in the selec-tion of search filters for specific tasks by identify-ing those filters that have been developed usingmethods which are liable to minimise bias andwhich have been tested in terms of their ability toretrieve relevant records with the minimal propor-tion of irrelevant records. Searchers and research-ers need to be aware of the process and limitationsinvolved in search filter development so that they
are able to make informed decisions about whetherto use them.In 2011, the Medical Research Council (MRC),
in partnership with the (now) National Institutefor Health and Care Excellence (NICE) announceda programme of funding for methodologicalresearch to underpin NICE decision-making.Given the importance of evidence identificationfor the work of NICE in producing guidance andguidelines, and the general desire for standardisa-tion in research approaches, we developed a pro-posal to explore filter use. The objectives of thisresearch project were to improve our understand-ing of filter use by searchers and researchers, tolearn how searchers and researchers choose searchfilters and what information they would like toreceive to inform their choices. The proposal wasfunded as MRC research grant G090149612 (May2010–May 2012), and the research was under-taken by information specialists and researchersfrom the York Health Economics Consortium atthe University of York, the UK Cochrane Centre(now based at Lefebvre Associates Ltd), the Uni-versity of Aberdeen, and Healthcare ImprovementScotland.The research involved a multi-method approach:• Five literature reviews were carried out toinvestigate different aspects of performancemeasurement in search filters and diagnostictest accuracy studies (to which they are analo-gous), their reporting and the selection ofsearch filters by searchers and researchers;
• Interviews were carried out and a web-basedquestionnaire was made available to gaininformation on current filter use;
• The reviews, interviews and questionnaire sur-vey then informed the development of pilotfilter performance visualisations and guidanceon gathering and reporting search filter perfor-mance.
This study reports the findings from the inter-views and questionnaire survey which were con-ducted to improve the project team’sunderstanding of how searchers and researcherschoose and use search filters and also to under-stand what information searchers and researcherswould like to receive to inform their choice ofsearch filters.
Choosing and using methodological search filters, Beale et al.2
Methods
Study design
Phase I: Semi-structured interviews. The fundingfor this research was awarded as part of a pro-gramme to benefit NICE working practices. In thelight of that key audience for this research, theviews of NICE information specialists andresearchers seemed especially important for theproject. Information specialists and Project Manag-ers working for NICE, the NICE collaboratingcentres and the NICE Evidence Review Groups(ERGs) were invited to take part in a telephoneinterview. Each interview was recorded and tookno more than 45 minutes. After each interview, anemail containing a summary of the key pointsraised during the interview was sent to each inter-viewee. The interviewee was offered the opportu-nity to check the notes for accuracy and to addany additional points that might have occurred tothem after the interview had ended.
Phase II: Questionnaire survey. A questionnairesurvey was developed to obtain information on awider community of searchers’ knowledge of anduse of search filters. This was intended to benefitNICE through providing information on howother searchers might use, choose and amendsearch filters. The questions in the survey werebased on the findings from the reviews, and theinterviews that had already been undertaken aspart of this project. A web-based questionnairecomprising 23 questions was developed (Appen-dix).The questionnaire was made available on the
YHEC website (www.yhec.co.uk). NICE informa-tion specialists and NICE Project Managers wereinvited to complete the questionnaire, and it wasalso used to collect the views of the wider system-atic review, health technology assessment andguidelines information community via invitationssent out on seven email lists:• Lis-medical (1523 individuals belonging to anopen discussion list for members of the UKmedical & health care library community andother interested information workers);
• [email protected] (204 subscribersbelonging to the open discussion list of the
Cochrane Information Retrieval MethodsGroup);
• [email protected] (subscribers are informa-tion specialists who work for the evidencereview groups providing services to NICE);
• [email protected] (subscribersare information specialists who are membersof the HTAi organisation);
• Cochrane TSCs (100 members of the Cochra-ne Trials Search Coordinators email discus-sion list);
• [email protected] (1000 mem-bers of the discussion list of the EuropeanAssociation for Health Information andLibraries).
Potential respondents were also alerted to thequestionnaire via posts on the YHEC Facebookpage, and messages sent out via the YHEC Twitteraccount. The survey was available for completionduring a 4-week period (22 July 2011–18 August2011), with email reminders being sent out 1 weekbefore the final deadline to maximise the responserate.
Results
Data collection
Twelve interviews were conducted in May andJune 2011. Ten of the interviews involved oneinterviewee, one interview involved two intervie-wees and another interview involved four intervie-wees. In total 16 people took part in theinterviews. Fifteen of the individuals were inter-viewed by telephone whilst one was interviewedface-to-face.A total of 90 questionnaires were completed. It
is not possible to calculate a response rate as indi-viduals may have been members of more than onediscussion list. Furthermore, the number of respon-dents who were alerted to the questionnaire via theTwitter or Facebook messages is unknown. Justover three quarters of respondents (70/90; 77.8%)worked directly in information or library services,while the remaining respondents came from arange of backgrounds and included a consultant
Choosing and using methodological search filters, Beale et al. 3
physician, an associate scientist, a medical docu-mentalist and a learning resources officer. Thequestionnaire was completed by experiencedsearchers, with nearly half (44/90; 48.9%) having11 or more years of database searching experience.Additionally, three quarters of questionnairerespondents (68/90; 75.6%) reported that theydeveloped searches at least once a week, half ofwhom indicated that they developed searchesdaily.All but one respondent used MEDLINE, and most
(84/90; 93.3%) used The Cochrane Library data-bases. About three quarters used EMBASE (70/90;77.8%) and CINAHL (67/90; 74.4%), whilst a lit-tle under two-thirds (56/90; 62.2%) used Psy-cINFO.
When do researchers use search filters?
Interviewees. Findings from the interviews sug-gested that search filters are only rarely used byNICE information specialists. The reasons givenvaried according to the NICE output and are asfollows:• Clinical guidelines: For short clinical guide-lines, the team only use search filters on therare occasions when the search is restricted toa particular study design.
• Diagnostic guidance: The filters available donot work well.
• Interventional procedures guidance: Searchesare carried out at the point in time when (orbefore) products get a Conformit�e Europ�eenne(CE) mark, (a symbol given to products thatconform with relevant EU health and safetydirectives) and tend to be internet based as, atthis stage, there are few publications and fewpublications in databases.
• Technology appraisals:o Single Technology Appraisals involve a
review of the literature reviews submitted toNICE by the manufacturer and the ERG’stask is only to develop searches to test thevalidity of the searches carried out by themanufacturer;
o Multiple Technology Appraisals are popula-tion, intervention, comparison and outcome(PICO) driven and do not require methodsfilters.
Interviewees reported, however, that they some-times used search filters under the following cir-cumstances:• When carrying out searches for a particularstudy type;
• To help focus the question in order to achievea manageable numbers of records;
• On searches for small projects with limitedresources;
• To identify economic evidence.
Questionnaire respondents. Eighty-five of 90(94.4%) respondents indicated that they had usedmethodological search filters, and 5/90 (5.6%)indicated that they had not (question 6).About three quarters of respondents said that
they use search filters for extensive searches toinform guidelines or systematic reviews (question7), while just over half said they would use themfor rapid searches to answer brief questions and asimilar number would use them for scopingsearches to estimate the size of the literature on atopic (Table 1).Other reasons that were given for using search
filters included:• To reduce results to a manageable size/narrowdown results;
• To begin to identify MeSH and text words touse in developing a strategy;
• If the customised limits provided by the data-bases cannot be relied upon;
• To monitor trends.
Table 1 Circumstances in which questionnaire respondents
Choosing and using methodological search filters, Beale et al.4
Benefits of using filters (question 19) includedthat filters help focus results (38/90; 42.2%), aretried and tested (17/90; 18.9%), save time (9/90;10%), and offer transparency and consistency (5/90; 5.6%). Other benefits cited by individualsinclude helping to estimate workload in projectplanning and offering a ‘conceptual mapping ofthoughts’.Limitations of using filters (question 20) were
identified by respondents as the possibility ofmissing relevant studies (34/90; 37.8%), that filtersmight not always be fit for purpose (20/90;22.2%), there may be difficulties in appraising thefilter (9/90; 10%) and that filters are only as goodas their developer or the database indexing (13/90;14.4%). Other limitations cited by single respon-dents were that filters may produce too manyresults, there is a lack of instructions on how tocite the filters, it can sometimes be difficult tochoose between filters, and filters are sometimeschosen as a compromise.
How do searchers identify search filters?
Interviewees. The most frequently reported meth-ods used by interviewees to identify search filterswere:• The InterTASC information specialists’ Sub-Group (ISSG) Search Filters Resource3;
• Posting messages to discussion lists;• Consulting the Cochrane Handbook;
• Discussion with colleagues;• Using an in-house search manual;• Reviewing previous project methods.
Questionnaire respondents. Respondents searcheda range of sources to identify search filters (ques-tion 10). The most frequently reported sources forstudy design filters were those produced by theMcMaster Hedges Team. These filters are alsoincluded in many interfaces to MEDLINE. Furtherdetails of the seven most reported resources areprovided in Table 2.When describing which filters they currently use
(question 11), respondents cited search filters forRCTs (67/90; 74.4%) and systematic reviews (61/90; 67.8%) most frequently. The most frequentlycited filters in a specific topic were the Cochranefilters to identify RCTs (33/90; 36.6%).When asked how they keep up to date with
search filters (Question 15), the majority of respon-dents (60/90; 66.67%) indicated that they readjournal articles, 33.33% also used information pro-vided by managers or a work colleague, 50/90(55.55%) used websites, 67/90 (74.44%) used pro-fessional development meetings and trainingevents, 57/90 (63.33%) used email lists and 24/90(26.66%) used current awareness services. Emaillists cited included canmed-lib, clinlib, cochraneinformation retrieval methods group, cochrane tri-als search coordinators, eahil-l, evidence-basedhealth, [email protected],
Table 2 Search filter sources reported by questionnaire respondents
Resource
Study design
RCTs
Systematic
reviews DTAs Prognosis Aetiology
Other
trials
Cochrane (this term is used to capture references to Cochrane
reviews, the Cochrane Handbook, the Cochrane Collaboration
Choosing and using methodological search filters, Beale et al. 5
expert searching, hslg, intertasc information spe-cialists subgroup, isg-information resources, lib-helix, lis-medical, lis-nursing, local health librariesnetwork, medlib, ncc-information-specialists,[email protected], webenz.
How do researchers decide which search filters
to use?
Interviewees. Sensitivity and precision wereimportant to most interviewees. Several intervie-wees indicated that they would test the filtersagainst a target set of references to see which filteridentified most references, although the approachesdescribed tended to be pragmatic in nature. Otherapproaches included:• Using the most up to date filter;• Considering the credentials of the people/orga-nisation that developed the filter;
• Assessing the methods used to develop the fil-ter;
• Reviewing details about the filter published inthe academic literature;
• Taking advice from senior colleagues.
Questionnaire respondents. When asked abouthow they typically use search filters, about four-fifths of respondents (73/90; 81.8%) indicated thatthey use different filters depending on whethertheir search needs to be sensitive or precise (ques-tion 9).Respondents were asked how they decide which
filter to use by selecting one, or more, optionsfrom a list (question 12). Just over half of therespondents reported that they generally use the fil-ters that are already available in the database beingsearched (48/90; 53.3%) (Table 3), rather than typ-ing in another filter.Other approaches used by respondents to help
them decide which filters to use were trial anderror, or comparing results and comparing filters’sensitivity and precision.Respondents were asked how, when faced with
a choice of methodological filters, they chosewhich one to use (question 21). Fifteen respon-dents (15/90; 16.7%) said that they required adescription of the filter, 25 (25/90; 27.8%) requiredinformation on the performance of the filter interms of validation, while 18 (18/90; 20%)
required information on sensitivity and precision.Ten respondents (10/90; 11.1%) required informa-tion on who had published the filter and six saidthey would ask colleagues for advice.When asked about information which had
helped with filter selection in the past (question16) similar responses were received to question 21but in addition single respondents recorded the fol-lowing observations:• Personal knowledge or choosing the one thatlooks logical;
• Length – ‘the shorter the better’;• Focus;• Flexibility/modifiability;• Testing against a target reference list;• InterTASC ISSG website;• Assessing the search terms within the strategy.Once they had found a search filter, just over
half of the respondents (50/90; 55.6%) reportedthat they sometimes amend search filters (apartfrom adding a subject search), while four respon-dents (4/90; 4.4%) indicated that they alwaysamend search filters, a third (30/90; 33.3%) do notamend them (question 13), while six individualsdid not respond to this question. Twenty-sixrespondents (26/90; 28.9%) indicated that theyamended filters to improve sensitivity and/or speci-ficity (question 14), and thirty respondents (30/90;33.3%) indicated that they amended search filtersby adding or removing subject headings, textwords or both. Other methods used to amendsearch filters mentioned by respondents included:• Adapting the filter to another database (n = 1);
Choosing and using methodological search filters, Beale et al.6
• Looking at adjacency or truncation (n = 6);• Examining which lines of syntax are produc-ing zero, or too many results (n = 1);
• Weighting word algorithms (n = 1);• Taking advice from other information special-ists (n = 1).
Nearly, all of those respondents who amendedsearch filters (45/54; 83.3%) tested the effect ofthe amendment (question 14) by either comparingthe results with and without the filter, or bywhether known relevant papers had been identi-fied. Three quarters of respondents (68/90; 75.9%)documented their amendments when they wrote upthe searches, using diverse approaches includingreproducing the entire search string with a writtensummary and rationale for changes and effects,keeping detailed spreadsheets and notes on strat-egy changes and including narratives on filteramendments in the Methods section of the review.
What information would help researchers
choose between filters?
Interviewees. Some interviewees expressed a pref-erence for a simplified measure of search filter val-idation, such as a rating system. This was anexpression of their lack of time to fully criticallyappraise the methodological robustness of filterdesign. One interviewee stressed that they wouldneed to have confidence in whoever carried outthe rating. Other interviewees said that a summarydocumenting sensitivity and precision would beuseful, although it was recognised that this mightbe subjective.
Questionnaire respondents. In the 56 free-textanswers supplied in response to question 21,respondents indicated that information on searchfilter performance measures such as validation (25/90; 27.8%), sensitivity and precision (18/90; 20%)and a description of the filter (15/90; 16.7%) hadhelped them to choose which filter to use. Otherinformation identified as useful for selecting a fil-ter included:• Results of own testing (11 respondents);• Colleague recommendations/discussion (6respondents);
• Knowledge that the filter is appropriate for arelevant database (4 respondents);
• Knowledge of the creator/developer (3 respon-dents);
• Simplicity/easy to understand (3 respondents);• Ease of use (including automatic loading) (2respondents).
Respondents reported that the main factorswhich would make choosing a filter easier werethe availability of a critical appraisal or evaluation(16/90; 17.7%) and more information (15/90;16.6%), including the effectiveness of the filter,what it does or what it provides, what it excludes,its limitations, when it was last updated, advanta-ges and disadvantages, sensitivity and precision,and what testing had been completed. Respondentsalso reported that they wanted to be confident inthe author/developer (10/90; 11.1%) and that hav-ing the filter available in a central storage locationwas an important consideration (7/90; 7.7%).Individuals also suggested that the following
changes would make choosing between search fil-ters easier:• Improved labelling/indexing of articles so theymight be more easily retrieved (1 respondent);
• More up to date coverage on the ISSG SearchFilter Resource website (1 respondent);
• The need for more ‘professional noise’ abouta new filter (1 respondent).
Discussion
In 2004, Jenkins and Johnson reported that,although researchers were aware of filters, therewas a low level of usage.2 Since then, it appearsthat more people are using filters to inform theirresearch, and filters are being used for a range ofsearching tasks.The present study has several limitations.
Although we do not know what proportion ofsearch filter users, we have reached, questionnaireanalyses show that our sample includes librariansand other researchers involved in supporting sys-tematic reviews, technology appraisals and guide-line development, all of whom represent our targetaudience. From the mailing lists of which respon-dents report being members, we can tell that manyare information specialists supporting the produc-tion of health technology assessments, guidelinesand systematic reviews. We therefore expect theresults to be broadly generalisable to such
Choosing and using methodological search filters, Beale et al. 7
librarians and other researchers. The mailing listswe circulated had at least 2857 subscribers, whichdoes however mean that the 90 respondents repre-sent a low response rate. The mailing lists we usedranged from lists with high proportions of informa-tion specialists with roles similar to NICE informa-tion specialists to more general lists where many ofthe members might not routinely use search filters.In terms of in depth interviews, our sample was
limited to NICE staff, for whose benefit theresearch was being undertaken. It would be usefulfor future research to interview searchers andresearchers from other settings, to understandwhether the NICE experience is generalisable.The questionnaire we developed was quite
lengthy and, in retrospect, might have benefitedfrom being shorter. However, the response rate toearly questions is similar to that of later questions,suggesting that the length of the questionnaire wasnot a deterrent to any of the individuals who actu-ally submitted a response. It might have helped toachieve more standardised results and fewerambiguous answers if we had given respondentsmore multiple choice questions. Respondentsdescribed resources quite vaguely at times andsometimes the same resource was described usingseveral different names. We have made someassumptions about the variant naming of widelyused resources that we have applied to provide amore succinct report and to ensure that the mostfrequently reported resources are identified assuch. We have not, however, corrected what maybe ‘errors’ in the responses, for example wherecertain filters may be incorrectly described orascribed to the wrong author or organisation.
When do searchers and researchers use search
filters?
The awareness and use of search filters seems tohave developed considerably in the decade sinceJenkins’ paper.2 Most respondents seem to knowwhere to look for filters from well-established pro-ducers and collections. However, the responsesdemonstrate a wide variation in the confidencewith which questionnaire respondents choose fil-ters. There are also contradictions between the dif-ficulties respondents express in terms of selectingbetween filters (acknowledging the possible com-
plexities of filter design) and the commonplacepractice of searchers adapting published strategiesto fit their own requirements (ignoring the fact thatmany filters are designed to perform in a quitespecific way). Several respondents have developedtheir own filters for local use.The responses indicate that search filters are
used more frequently for large-scale reviews andslightly less often for simpler scoping and rapidsearches. This may reflect different practices inscoping and rapid searches because fewerresources will be searched and less sensitive sub-ject searches will be employed because of the lim-ited timescale. Adding a filter to an alreadyfocused search might be seen as risking missingstudies. For all types of searches, search filtersoffer an opportunity to focus the numbers ofrecords retrieved, which can be helpful when timeis limited. Search filters are predominantly viewedby respondents as a tool to maximise sensitivityrather than precision (although this is not theintended objective of all filters), but seem to beused to achieve optimal sensitivity and precision.
What information would help researchers
choose between filters?
The responses to the questionnaire have manymessages for search filter designers. Filter perfor-mance measures need to be signposted moreclearly and succinctly to help searchers make bet-ter use of available filters. Filter and websitesdesigners should present less information (to avoidinformation overload) and to ensure that perfor-mance information can be clearly seen. Respon-dents also reported that they wanted to beconfident in the author/developer. While the prove-nance of filters is clearly important to some search-ers, there are no established parameters to measurethis confidence. Clear authorship labelling and pro-vision of detailed methods to show the robustnessof the development methods would not only assistusers of filters but would also help filter designersachieve recognition for their filters. The conve-nience of having filters by well-established produc-ers available within database interfaces (such asthe PubMed Clinical Queries) encourages theiruse. However, the most convenient search filtersmay not always be the best for the task and
Choosing and using methodological search filters, Beale et al.8
searchers and researchers need to know how tochoose when a range of sensitive, precise or opti-mal strategies are offered.Respondents require more information on the
validation of search filters. They value and useresources such as the ISSG Search Filter Resourceand the filters of the McMaster Hedges Team. Theformer provides a listing of filters in one place bystudy design, which has a convenience factor. Thelatter provide search filters developed with docu-mented methods within database interfaces and are‘badged’ with the authority of both the researchteam and the National Library of Medicine. In con-trast to the methodological and publisher qualityseals of the McMaster filters, the BMJ Clinical Evi-dence and SIGN contain little information on filterproduction and/or validation. However, the filterson these websites seem to be widely used, suggest-ing that the authorship is the seal of quality.Respondents did not necessarily feel that all
their requirements were currently being met. Theywould like translations of filters for different data-bases and interfaces, more strategies independentof indexing language (to facilitate transferral acrossdatabases) and filters for a wider range of studydesigns and other topics. This provides a researchagenda for any search filter authors willing to takeup the challenge.Respondents keep informed about developments
in search filters through a wide variety of methodsand resources, which suggests that search filter andwebsite designers face a marketing challenge. High-lighting new filters to key audiences such as infor-mation specialists and systematic reviewers byinclusion in resources such as the Cochrane Hand-book and the ISSG Search Filter Resource wouldhelp to promote new filters beyond the simple pub-lication of a journal paper. In addition, a large num-ber of email lists are used for current awareness andpromotion of new filters through those lists wouldseem an efficient way to reach potential users.Although the use of search filters seems to be
quite widely documented, and amendments notedin search reports, there seems to be scope for pro-moting clarity around the use, and amendment, ofsearch filters. This, again, is an issue for filterauthors and website producers. There is clearly alarge amount of ad hoc filter amendment beingundertaken: searchers take filters and adapt them
for their own purposes. This would seem to indi-cate a lack of awareness that the filters may bedesigned for a purpose, or arrived at after exten-sive exploration to justify the use of specific terms.The performance assessment of amended searchfilters does not seem to be a priority for manysearchers. Filter developers should consider howthey want their filters to be used and perhapsattach guidance or caveats to the filters. Guidancefor filter adaptation may also be merited so that fil-ter developers are credited for the original workbut absolved from the effects of the adaptations.Many filter developers retain their gold standardsand might be willing to test adaptations.The original impetus for many search filters was
to maximise sensitivity, but increasingly, possiblydue to limited resources, searchers seem to bedemanding improvements in precision. Future filterdevelopments (for interfaces which use Booleansearching) need to continue to improve precision.The advent of full text searching and semantic analy-sis of both full text and bibliographic records maysee filters used in different ways in future. For exam-ple, sensitive filters might be used to identify recordsfrom databases. These results might then be pro-cessed using semantic analysis software trained toidentify records of specific types. The latter approachwill have search algorithms (filters) that are morelike semantic rules than the dichotomous (relevant/not relevant) search filters we see in interfaces tobibliographic databases such as PubMed. Recentsearch filters have used textual analysis approachesin their design.6,19 The extent to which textual analy-sis alone can be relied upon in the future to distin-guish relevant records from irrelevant records isunder investigation.19,20 When using textual analysisapproaches the onus will be on the searcher to selectthe performance levels, that is, an acceptable proba-bility of a record being relevant is acceptable.
Future research
Areas for future research could include presentingsearch filter performance in different ways andtesting searchers’ understanding of the filter perfor-mance trade-offs offered. It would also be helpfulto gain more information on exactly how searchfilters are amended in practice, in order to informfilter design: filter designers assume searchers want
Choosing and using methodological search filters, Beale et al. 9
sensitive filters, or optimised filters, but in factsearchers may prefer different options or to be ableto choose using a sliding scale of sensitivitydepending on the number of records retrieved.
Conclusion
Search filters are used mainly for reducing largeresult sets (introducing focus) and assisting withsearches that are focused on a single study type.Searchers use several key resources to identifysearch filters, but may find choosing between filtersproblematic. Features that would help with choicesinclude making information about filters less techni-cal, offering ratings and providing more detail aboutfilter validation strategies and filter provenance.
Acknowledgements
This research was funded as part of the UK Medi-cal Research Council grant G0901496. We wouldlike to thank all the interviewees and questionnairerespondents for their time and their thoughtful andconsidered contributions. We are also grateful forthe comments of the anonymous referees whoreviewed this paper.
References
1 Jenkins, M. Evaluation of methodological search filters–areview. Health Information and Libraries Journal 2004, 21,148–163.
2 Jenkins, M. & Johnson, F. Awareness, use and opinions ofmethodological search filters used for the retrieval of evi-dence-based medical literature-a questionnaire survey.Health Information and Libraries Journal 2004, 21, 33–43.
3 The InterTASC Information Specialists’ Sub-Group. The In-terTASC Information Specialists’ Sub-Group Search FilterResource. York: InterTASC Information Specialists’ Sub-Group; [cited 2014 30 January]; Accessible at: https://sites.google.com/a/york.ac.uk/issg-search-filters-resource.
4 National Library of Medicine. PubMed Clinical Queries. Mary-land: National Library of Medicine; [cited 2014 30 January];Accessible at: http://www.ncbi.nlm.nih.gov/pubmed/clinical.
5 White, V. J., Glanville, J., Lefebvre, C. & Sheldon, T. A. Astatistical approach to designing search filters to find system-atic reviews: objectivity enhances accuracy. Journal ofInformation Science 2001, 27, 357–370.
6 Glanville, J. M., Lefebvre, C., Miles, J. N. V. & Camosso-Stefinovic, J. How to identify randomized controlled trials inMEDLINE: ten years on. Journal of the Medical LibraryAssociation 2006, 94, 130–136.
7 Haynes, R. B., Wilczynski, N., McKibbon, K. A., Walker, C.J. & Sinclair, J. C. Developing optimal search strategies fordetecting clinically sound studies in MEDLINE. Journal of theAmerican Medical Informatics Association 1994, 1, 447–458.
8 Wilczynski, N. L., Morgan, D., Haynes, R. B. & Hedges, T.An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Medical Informaticsand Decision Making 2005, 5, 20.
9 Boynton, J., Glanville, J., McDaid, D. & Lefebvre, C. Iden-tifying systematic reviews in MEDLINE: developing anobjective approach to search strategy design. Journal ofInformation Science 1998, 24, 137–154.
10 Bak, G., Mierzwinski-Urban, M., Fitzsimmons, H., Morrison,A. & Maden-Jenkins, M. A pragmatic critical appraisal instru-ment for search filters: introducing the CADTH CAI. HealthInformation and Libraries Journal 2009, 26, 211–219.
11 Glanville, J., Bayliss, S., Booth, A., Dundar, Y., Fernandes,H. & Fleeman, N. D. So many filters, so little time: thedevelopment of a search filter appraisal checklist. Journal ofthe Medical Library Association 2008, 96, 356–361.
12 Lefebvre, C. Methodological search filter performance: assess-ment to improve efficiency of evidence information retrieval.MRC research grant G0901496. London:MRC, 2010.
13 Centre for Reviews and Dissemination. Centre for Reviewsand Dissemination. York: Centre for Reviews and Dissemi-nation; [cited 2014 28 January]; Accessible at: http://www.york.ac.uk/inst/crd/.
14 Health Information Research Unit. Hedges. Hamilton, ON:McMaster University Health Information Rearch Unit; [cited2014 28 January]; Accessible at: http://hiru.mcmaster.ca/hiru/HIRU_Hedges_home.aspx.
16 National Library of Medicine. PubMed. Maryland: NationalLibrary of Medicine; [cited 2014 28 January]; Accessible at:http://www.ncbi.nlm.nih.gov/pubmed.
17 National Institute for Health and Care Excellence. Journalsand Databases. London: National Institute for Health andCare Excellence; [cited 2014 28 January]; Accessible at:https://www.evidence.nhs.uk/about-evidence-services/journals-and-databases.
19 Hausner, E., Waffenschmidt, S., Kaiser, T. & Simon, M.Routine development of objectively derived search strate-gies. Systematic Reviews 2012, 1, 19.
20 Glanville, J., Lefebvre, C., Porter, B. & Negosanti, P.Improving search efficiency for economic evaluations inmajor databases using semantic technology. The CochraneColloquium; Colorado, USA. October 2010.
Choosing and using methodological search filters, Beale et al.10
Questionnaire
1. Please state your job title:__________________________________________
2. How long have you been searching databases such as MEDLINE (years)?
3. How often do you develop new search strategies as part of your work (For example searches to find treatments for condition)?NoneDailyOnce a weekOnce a monthLess than once a month
4. What types of searches do you carry out (please tick all that apply)Rapid searches to answer brief queriesScoping searches to estimate the size of the literature on a topicExtensive searches to inform guidelines or systematic reviewsOther
If Other, please describe in the box below
5. Which databases do you search regularly?MEDLINEEMBASECINAHL PSYCINFOCOCHRANE LIBRARY databases (CDSR, DARE, NHS EED, CENTRAL, HTA)HEED
Please list any other databases that you use regularly in the box below
6. Methodological search filters (also known as Clinical Queries or Search Hedges) are used to find specific study designs such as randomised controlled trials. Have you ever used methodological search filters? Yes/No
7. In what circumstances would you use methodological search filters?Rapid searches to answer brief queriesScoping searches to estimate the size of the literature on a topicExtensive searches to inform guidelines or systematic reviewsOther
If Other, please describe below
8. Do you always use a filter when providing searches for similar types of projects? (For example, if you were searching for randomised controlled trials in MEDLINE would you always use a methodological search filter?) Yes/No
If No, please provide details about the circumstances when you would not use a filter?
Choosing and using methodological search filters, Beale et al. 11
9. Please select the statement which describes your typical practice:I use different search filters depending on whether my search has to be sensitive or preciseI use the same search filter irrespective of the focus of the search
10. If you had to find a methodological search filter for a specific study design, where would you look?
11. What methodological search filters do you use at present?
Randomised controlled trials - please list the author or name of each of the filters you use?
Systematic reviews - please list the author or name of each of the filters you use
Diagnostic studies - please list the author or name of each of the filters you use
Studies of prognosis - please list the author or name of each of the filters you use
Studies of etiology - please list the author or name of each of the filters you use
Other trials - please list the author or name of each of the filters you use
Guidelines - please list the author or name of each of the filters you use
Economic evaluations - please list the author or name of each of the filters you use
Other study methods - please list the author or name of each of the filters you use
Choosing and using methodological search filters, Beale et al.12
12. How do you decide which filter to use? Please select all which applyCustom and practice – I’ve always used the same filtersGuidance from a colleagueI research the available filters and chose the best for my purposesI follow standing operating procedures/guidance on filters provided by my organisationI use international/national guidance on best practiceI use the filters available in the database interfaces I use, e.g. Clinical Queries
Please provide details on any other approaches you use to decide which search filter to use
13. Apart from adding a subject search, do you amend methodological search filters?NoSometimesAlways
14. Please can you provide us with some more information about amending search filters?
Why, typically, do you amend search filters?
How do you amend search filters?
Do you test the effects of any amendments you make? Yes/No
If Yes, how do you test the amendments?
Do you document the amendments when you write-up your searches? Yes/No
If Yes, how do you document the amendments?
15. How do you keep up to date with methodological search filters? (Please tick all the apply)Reading journal articlesCurrent awareness services
Please list typical current awareness services that you use
