EVALUATION OF CLIENT SATISFACTION IN A COMMUNITY … · la participation communautaire lors de...

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 97The Canadian Journal of Program Evaluation Vol. 17 No. 1 Pages 97–117ISSN 0834-1516 Copyright © 2002 Canadian Evaluation Society

EVALUATION OF CLIENT SATISFACTIONIN A COMMUNITY HEALTH CENTRE:SELECTION OF A TOOL

Vickie BoechlerAnne NeufeldFaculty of NursingUniversity of AlbertaEdmonton, Alberta

Robert McKimNortheast Community Health CentreEdmonton, Alberta

Community health centres (CHCs) have emerged as sites inwhich services based on a primary health care (PHC) model aredelivered. In the evaluation of the newly established NortheastCHC in Edmonton, Alberta, client satisfaction was a significantpart of the evaluation component on community participation.A reliable and valid tool was sought to measure the dimensionsof client satisfaction appropriate for a CHC that is based on theprinciples of PHC. Using criteria established to assist the evalu-ators in choosing an appropriate tool, the Service SatisfactionScale was selected because it was comprehensive, appropriate,and psychometrically acceptable. Refinement on the basis of cli-ent and provider input led to a revised client satisfaction toolthat is currently being tested. The authors suggest that, con-gruent with the PHC model, a multimethod approach that in-corporates focus groups and individual interviews should beemployed in evaluation of client satisfaction to add useful in-formation about client perspectives.

Les centres de santé communautaire (CHC) offrent des servi-ces fondés sur un modèle de soins de santé primaire (PHC). Lasatisfaction de la clientèle était une composante importante pourla participation communautaire lors de l’évaluation du nouveauCHC Nord-Est, Edmonton, Alberta. Un instrument valide et fia-ble a été recherché afin de mesurer les dimensions de satisfac-tion de la clientele appropriées pour un CHC fondé sur desprincipes de soins de santé primaire. Tenant compte des critè-res établis pour les évaluateurs afin de choisir un instrumentapproprié, le «Service Satisfaction Scale» (Attkisson &Greenfield, 1994; Greenfield & Attkisson, 1989) a été choisi car

Abstract:

Résumé:

THE CANADIAN JOURNAL OF PROGRAM EVALUATION98

il est compréhensif, approprié et psychométriquement accepta-ble. L’instrument a été revisé selon les commentaries des pour-voyeurs et des clients. Une évaluation de l’instrument reviséest présentement en cours. Les auteurs suggèrent une appro-che multi-méthodes, congruente avec le modèle PHC, qui de-vrait comprendre des groupes de consultation ainsi que desentrevues individuelles afin d’évaluer la satisfaction de la clien-tèle. Cette approche apporterait des renseignements utiles surla perspective des clients.

Community health centres (CHCs) in Canada haveemerged as a model of service delivery and health care managementthat is based on the principles of primary health care. CHCs arenon-profit health care organizations or associations that offer abroader range of services than conventional medical practices(Church & Lawrence, 1999). Objectives of CHCs include (a) foster-ing the empowerment and health of the individual and the commu-nity; (b) improving access to appropriate primary care services;(c) promoting health and preventing illness; (d) developing servicesthat maximize availability of service providers and resources; and(e) promoting a multidisciplinary team approach in meeting clientsocial and health care needs (Church & Lawrence, 1999). Programsand services offered may include health promotion and illness/in-jury prevention, mental health services, community development,and outreach community-based services. CHCs employ multi-disciplinary teams of health care professionals and give priority toallocation of resources to meet the specific needs of the populationthey serve.

Evaluation of primary health care services in CHCs can contributeto program improvement, increased program efficacy and efficiency,and cost-effectiveness in a climate of scarce resources. Similarly,evaluation findings are often required to ensure future funding fromregional health authorities or to secure special project grants fromdifferent levels of government. Evaluation also serves to demonstratethe overall effectiveness of the CHC model in relation to other typesof service delivery within regional health authorities.

Client satisfaction is an important component of evaluation of allCHC services because satisfaction survey results provide informa-tion about the quality of multidisciplinary care from the perspec-tive of clients (Davis & Hobbs, 1989). Pascoe (1983) argues that clientsatisfaction serves as an outcome measure of the quality of healthcare and provides a consumer perspective that contributes to a com-

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 99

plete, balanced evaluation of services. Client feedback alerts themultidisciplinary health care team to clients’ needs, concerns, andperceptions of service and is useful for improving existing programsand developing new ones. Evaluation of client satisfaction also re-assures clients that their concerns are important and provides amechanism for participation that can foster empowerment.

Clients who are satisfied may also be more willing to adhere to atreatment regimen. As well, satisfied clients are loyal and may becounted on for return business and referrals (Steiber & Krowinski,1990). Retention of clients who might otherwise avoid health careor seek alternate providers is important to the success and viabilityof the CHC. Health providers in any service that keeps clients satis-fied will see an improved image as a direct and immediate conse-quence (Steiber & Krowinski, 1990).

However, the measurement of client satisfaction is not without itsfaults. Haas (1999) has described a number of these shortcomings,including inability to capture a true expression of clients’ feelingstowards care because of their belief that they are not knowledge-able enough to comment on their care. Furthermore, clients are notusually asked to identify priorities among their beliefs. Finally, cli-ent satisfaction is only a partial measure of client outcome, yet pro-gram decision makers often see results as directly applicable tooutcome and often make resource decisions based on the results.Avis, Bond, and Arthur (1997) cite similar issues and describe thebias inherent in the clients’ desire to answer positively because theydo not want to offend their caregivers. They suggest that other, morequalitative measures may be a better way to address client satisfac-tion. Therefore, based on these observations, the evaluation teamrecommended that a survey be used in conjunction with focus groupsand individual interviews with clients who used the resources ofthe centre. The evaluation team comprised a group of external inde-pendent consultants who were engaged to complete the evaluationof the Northeast Community Health Centre, and an internal evalu-ator. Satisfaction was included as one of four domains of a majorformative evaluation at the centre. The evaluation was completedover the first 16 months of operation and was used to help shapethe evolution of the primary health care service delivery model atthe site.

One of the challenges in evaluating client satisfaction in a CHC isto identify appropriate measurement tools (Eriksen, 1995; Forbes


& Neufeld, 1997; Munro, Jacobsen, & Brooten, 1994; Pascoe, 1983).Well-defined criteria exist to guide researchers and evaluators inappropriate selection of survey tools (Burns & Grove, 1993). Thereare many client satisfaction tools available (Atlantic InformationServices, 1998/1999; McGee, Goldfield, Riley, & Morton, 1997); how-ever, there are no criteria specifically for selecting a client satisfac-tion tool for use within a CHC guided by the primary health care(PHC) model.

The purpose of this project was to use predetermined criteria to se-lect a client satisfaction tool. The tool must be useful for evaluationin a multiservice CHC using a PHC model.

BACKGROUND

Conceptualization of Client Satisfaction

Client satisfaction has been defined in a variety of ways (Eriksen,1995; La Monica, Oberst, Madea, & Wolf, 1986; Linder-Pelz, 1982;Pascoe, 1983; Risser, 1975; Strasser & Davis, 1991; Williams, Coyle,& Healy, 1998). Because the conceptual definitions of satisfactionare not consistent across measurement tools, comparisons amongstudies are generally not meaningful (Munro et al., 1994). Ware,Davies-Avery, and Stewart (1978) identified the gold standard forclient satisfaction measures as follows: accessibility/convenience,availability of resources, continuity of care, efficacy/outcomes of care,finances, humanness/interpersonal manner, information gathering,information giving, pleasantness of surroundings (physical environ-ment), and quality/competence. As these diverse dimensions indi-cate, client satisfaction is a multidimensional concept (Attkisson &Greenfield, 1994; Strasser & Davis, 1991; Ware et al., 1978), includ-ing dimensions that may either be specific to a health care settingand provider or cross boundaries between health care providers andsettings (Forbes, 1996). However, viability for generalizability andfuture use is lost when unique centre-specific definitions are usedinstead of variables derived from commonly identified concepts ortheories (Mahon, 1996). Hence, when one is reviewing the litera-ture for an appropriate tool to use to measure client satisfaction, itis imperative that the tools be considered only if their conceptualdefinitions are similar.

Some researchers claim that satisfaction and dissatisfaction areopposites poles of the same continuum (Harpole, Orav, Hickey,


Posther, & Brennan, 1996; Pascoe, 1983; Ware, 1981), whereas oth-ers argue that satisfaction and dissatisfaction exist as separate con-cepts (Comley & Beard, 1998; Eriksen, 1995; Forbes, 1996; La Monicaet al., 1986). Some have viewed the satisfaction construct as a cli-ent’s comparison between his or her antecedent expectations for serv-ice and what that person actually experiences (Linder-Pelz, 1982).Linder-Pelz further argued that client satisfaction was inverselyrelated to expectations; if expectations were low upon entering thehealth care system, then satisfaction would be greater; if expecta-tions were high, then client satisfaction would be lower. Even thoughsome clients may feel that the service has failed in its “duty,” theymay not evaluate it negatively or hold it responsible because theyaccept that there may be mitigating circumstances (Williams et al.,1998). High satisfaction ratings may reflect the attitude that “theyare doing the best that they can” and not necessarily that clientshave had good service or experiences. Agreement has not beenreached regarding the definitions, dimensions, boundaries, and de-terminants of satisfaction and dissatisfaction because of the com-plexity of the constructs (Forbes, 1996; Haas, 1999; Pascoe, 1983).

Contribution of Client Satisfaction in Evaluation

Attkisson and Greenfield (1994) maintain that knowledge of clients’reactions to service can make multiple contributions: (a) satisfac-tion findings may be used to improve services, training, financing,and organization; (b) perceptions of satisfaction may aid negotia-tion between the process of care and its consequences and help toillustrate what works; and (c) understanding the client’s “point ofview” may be useful in choosing providers or studying service utili-zation patterns. For example, studies involving primary health careclinics staffed with advanced-practice nurses (nurse practitionersor clinical nurse specialists) have reported the greatest client satis-faction. The reasons cited for greater satisfaction with improved serv-ice include better ability to negotiate with care providers andenhanced ability to choose priorities (Graveley & Littlefield, 1992;Hill, 1997) and the lowest cost per visit (Graveley & Littlefield, 1992).These findings can contribute to improved quality and economy indelivery of services.

Issues in Measurement of Client Satisfaction

Issues in measuring client satisfaction include determining whatdimensions or attributes should be measured, how the data should


be collected, reliability of the measurement tool, and whether to de-velop a unique tool or use a pre-existing one (Acorn & Barnett, 1999).Expression of satisfaction may not necessarily mean that a criticalevaluation has taken place; such expression may arise from lack ofan opinion or acceptance of medical paternalism (Williams, 1994).Williams argues that although some clients might critically evalu-ate their health care, most clients are very positive, allowing care tobe of extremely poor quality before expressing dissatisfaction. Be-cause populations differing in sociodemographic characteristics maynot value or measure providers and services in the same way (Ware& Davies, 1983), a standardized tool for measuring client satisfac-tion has not been readily identified (Forbes, 1996). Furthermore,being able to sample satisfied as well as dissatisfied clients is notalways guaranteed when doing surveys. Anonymous responses alsopose problems in interpretation when some surveys report negativesatisfaction and some report positive satisfaction. This issue of rep-resentation and interpretation again makes interpretation of resultsdifficult.

Reliability and validity. There is growing reliance on use of clientsatisfaction scales that have demonstrated reliability and validity(Attkisson & Greenfield, 1994; Eriksen, 1995; Greenfield &Attkisson, 1989; Ware & Davies, 1983). McIver and Meredith (1998)argue that there is no “off the shelf” questionnaire that will satisfythe requirement of eliciting client views in all service delivery ar-eas. However, ad hoc generation of client satisfaction tools specificto one organization leads to problems (Acorn & Barnett, 1999) andshould no longer be employed (McKinley, Manku-Scott, Hastings,French, & Baker, 1997). First, there is a potential to use a tool thatis not valid and reliable. Second, one is unable to compare findingsacross programs, organizations, and sites, and therefore findingsfrom similar evaluations cannot be shared.

Many issues remain unresolved in the area of client satisfaction.The main unresolved issue is the lack of conceptual clarity and con-sensus among those working in the area. Other issues include (a)the multidimensional nature of satisfaction, resulting in many at-tributes/dimensions to measure, and (b) variations in whether sat-isfaction and dissatisfaction are considered on the same continuumor viewed as different concepts. These early questions continue tochallenge researchers working in the area of client satisfaction (Aviset. al., 1997; Haas, 1999).


PROJECT DESCRIPTION

The setting for the project was the Northeast Community HealthCentre, located in Edmonton, Alberta. Client satisfaction was animportant component of multiple evaluations of a variety of CHCprograms occurring at the site. Like Strasser and Davis (1991), weview client satisfaction as a value judgment of and response to per-sonal health care encounters and the CHC environment, and thisdefinition guided our selection of an appropriate tool. The empiricalreferents or dimensions of satisfaction as defined by Ware et al.(1978), used by others (Greenfield & Attkisson, 1989; Linder-Pelz,1982; Strasser & Davis, 1991) and recurring in published client sat-isfaction tools, include reference to client, health care provider, andsetting characteristics. Within the context of the current project,the dimensions chosen as suitable for a CHC evaluation were asfollows: accessibility, convenience, availability of resources(multidisciplinary), continuity of care (provider), outcomes of care,humanness/interpersonal manner, information gathering and giv-ing (client education), and physical environment.

Using a research planning model similar to that of Davis and Hobbs(1989), we used the following steps to guide the choice of a clientsatisfaction measure for use in the evaluation of a CHC: (a) devel-opment of criteria to guide selection of client satisfaction tools forreview; (b) analysis of publications on client satisfaction that con-tain measurement tools and meet the selection criteria; (c) use ofthe criteria to establish priorities among satisfaction tools selected;and (d) selection of the final tool(s) to be recommended to the evalu-ation team for use in measuring client satisfaction at the CHC.

Development of Criteria for Selection of the Tool

A search of the literature for the years 1989–1999 was completedusing CINAHL, Medline, Healthstar, Eric, Embase, Canadian Re-search Index, and ProQuest databases. Based on this literature re-view and practical requirements for instrument use in evaluationsin a CHC setting, we chose the following criteria for selection ofpotential client satisfaction tools for review:

1. The researcher(s)/author(s) stated that the purpose of thetool was to measure client satisfaction and not some otherconstruct. Only client satisfaction dimensions used by Wareet al. (1978) and appropriate for use in a CHC were chosen.


For example, dimensions relating to inpatient experiencessuch as “food service” were not useful for a CHC withoutinpatient beds.

2. Psychometric characteristics were reported. As a minimum,reporting of reliability testing (internal consistency) andcontent validity testing was required. An alpha reliabilityof greater than .70 was required in order to ascertain thatthe items were tapping the same underlying concept or serv-ice dimension (Steiber & Krowinski, 1990).

3. The tool was acceptable to a population using a CHC (e.g.,easy to complete). In order to be congruent with the PHCservice delivery model, it was essential that the tool be ap-propriate and that the clients have had input regardingsuitable use of the tool. For example, clients will completesurveys incorrectly or refuse to participate if tools are com-plex. When questions are ambiguous or vague, clients arelikely to be confused (Steiber & Krowinski, 1990).

4. Time required to complete the questionnaire was feasible.Steiber and Krowinski (1990) recommended 15–20 minutesor between 45–75 questions for self-administered question-naires.

Articles were selected for further analysis that addressed at leasttwo of the identified criteria. Using this approach, we selected 13client satisfaction tools for further review. In addition to obtainingtools reported in the literature, we contacted other CHCs in Canadato procure their client satisfaction tools. Although some of the CHCshad locally developed client satisfaction tools, these had not beentested for reliability and validity. Several experts in the area of cli-ent satisfaction measurement were also contacted to obtain clientsatisfaction tools not published in the articles reviewed (Bear &Bowers, 1998; Eriksen, 1995; Larsson, Larsson, & Munck, 1998).All authors replied providing access to their tool.

Analysis of Publications that Contain Client Satisfaction MeasurementTools and Met the Selection Criteria

A summary of the 13 satisfaction tools selected for review is pre-sented in table format for ease of interpretation (see Table 1). Al-though each tool clearly stated that it was measuring clientsatisfaction, a definition of client satisfaction was cited in only 6 ofthe 13 tools (Eriksen, 1995; La Monica et al., 1986; Megivern, Halm,& Jones, 1992; Risser, 1975; Ryan, Collins, Dowd, & Pierce, 1995;


Table 1Table 1Table 1Table 1Table 1Preliminary Screening of Client Satisfaction Measurement ToolsPreliminary Screening of Client Satisfaction Measurement ToolsPreliminary Screening of Client Satisfaction Measurement ToolsPreliminary Screening of Client Satisfaction Measurement ToolsPreliminary Screening of Client Satisfaction Measurement Tools

Definition Congruence with CHC Number ofof Client Psychometric Population: Dimensions, Items and

Author Satisfaction Characteristics Setting, and Disciplines Type of Scale

Davis & Not defined Content validity as mea- 4 dimensions 20 itemsBush (1995) specifically sured by content experts • Psychological safety 5-point Likert

ranged from 0.75 to • Discharge teaching scale1.00 for scale items • Technical competence Anchors:Construct validity using of provider Completelyfactor analysis with • Information giving disagreesignificance set at 0.40 Emergency room CompletelyItems loading on more Measures satisfaction agreethan 1 factor — highest with nursing careloading used 4 factors,those congruent withCHC population werevalidatedCronbach’s α coefficient0.92

Eriksen (1995) Importance Cronbach’s α coefficient 2 dimensions 15 itemsof stating the Two factors were 0.93 • Art of care 7-point Likertdefinition of and 0.94 • Tangibles/ scalesatisfaction environments Anchors:

Inpatient setting ExpectationsMeasures satisfaction not met at allwith nursing care Way beyond

expectations

Greenfield & Not defined Content/ expert validity 4 dimensions defined 30 itemsAttkisson specifically Cronbach’s α coefficient by Ware (1981) 5 point Likert(1989); scores 0.80 to 0.88 • Practitioner manner scaleAttkisson & Across populations and skill Anchors:Greenfield • 0.87 practitioner • Perceived outcome Terrible(1994) manner • Office procedures Delighted

• 0.80 perceived • Accessoutcome Inpatient and outpatient

• 0.69 office procedures settings• 0.66 access Across discipline scale

Harpole, Orav, Not defined Not reported 3 dimensions 23 itemsHickey, specifically • Courtesy of office staff 5-point LikertPosther, & • Timeliness of care scaleBrennan (1996) • Communication with Anchors:

provider PoorAmbulatory care ExcellentMeasures “patient dissatis-faction with …” care providers(continued next page)




La Monica, Defined as Content validity from 3 dimensions 42 itemsOberst, Madea, congruence professionals and clients • Dissatisfaction 7-point Likert& Wolf (1986) between Construct validity reported • Interpersonal support scale

patients’ based on inverse relation- • Good impression Anchors:expectations ship of satisfaction scores Hospital Stronglyand their to negative mood states Measures satisfaction disagreeperceptions Cronbach’s α coefficient with nursing care Stronglyof actual 0.89 to 0.92 across factors agreecare and 0.92 for total instrumentreceived Authors caution that

validity of any indirectmeasure of satisfactionwith care may be difficultto determine.

Marsh (1999) Not defined Cronbach’s α for entire 4 dimensions 18 itemsspecifically scale was 0.92. • General satisfaction 5-point Likert

Construct validity was • Humaneness scaledone through factor • Quality Anchors:analysis • Access Strongly

Managed care inpatient disagreesetting StronglyMeasures satisfaction with agreecare from nurses and/orphysicians

Mayer, Cates, Not defined Not reported 14 dimensions 50 itemsMastorovich, specifically • Quality care 7-point Likert& Royalty • Skill, ER physician scale(1998) • Skill, ER nurse Anchors:

• Overall satisfaction Low• Overall respect to patient High• Wait time to see provider• Physician’s ability to

explain condition• Staff effort to keep family

informed• Staff ability to keep

patient informed• Likelihood of returning• Rapidity of evaluation by

triage nurse• Triage nurse sensitivity

to patient pain• Overall discharge process• Explanation by triage nurseEmergencyPhone interview(continued next page)




McKinley, Specific Content validity as 8 dimensions 32 itemsManku-Scott, regarding established by experts • Communication and 5-point LikertHastings, satisfaction Construct validity done management scaleFrench, & with off-hours through intercomponent • Doctor’s attitude Anchors:Baker (1997) care by correlations • Continuity of care Strongly

alternate Cronbach’s α coefficient • Delay until visit disagreephysician 0.61 to 0.88 by scale • Access to out-of-hours Strongly

care agree• Initial contact person• Telephone advice• Overall satisfactionOut-of-hours careMeasures satisfactionwith physician care

Megivern, Patient Content validity deter- 10 dimensions 37 itemsHalm, & perception of mined through content • Art of care 5-point LikertJones (1992) care provision experts • Technical quality scale

• Physical environment Anchors:• Availability Poor• Continuity Excellent• Efficacy/outcomes• Recognition of individual

qualities and needs• Reassuring presence• Promotion patient

autonomyIn-patient critical carepatients and their families

Munro, Not defined Cronbach’s α entire scale 2 dimensions 28 itemsJacobsen, & specifically 0.97 • Dissatisfaction 5-point LikertBrooten (1994) Validity was reported • Interpersonal support/ scale

based on: good impression Anchors:• Those receiving more In-patient women’s health Strongly

intensive nursing care clients disagreescored higher Measured satisfaction with Strongly

• Significant correlations nursing care agreebetween total score anditems measuring satis-faction with nursing care

• No significant correla-tion between total scoreand items measuringsatisfaction withphysician care(continued next page)




Risser (1975) Defined as the Cronbach’s α 0.89 to 3 dimensions 25 itemscongruency 0.91 • Technical-professional 5-point Likertbetween what Content validity based on relationship scalea patient expert opinion and client • Educational relationship Anchors:expects from input • Trusting relationship Stronglycare and what Community clinics disagreecare was Nursing and nursing care Stronglyactually in primary care settings agreereceived

Ryan, Collins, Defined as Content validity deter- 7 dimensions 94-item sur-veyDowd, & fulfillment of mined through expert • Information and educationPierce (1995) expectations, opinion • Patient values

needs for care Test-reliability agree- • Emotional supportfrom the ment between items • Continuity and transitionpatient’s was 92% • Involving family andperspective friends

• Physical comfort• Coordination of carePhone interview

Ware, Snyder, Defined as a Content validity deter- • Access to care 55 itemsWright, & patient’s rating mined through expert • Financial aspects 5-point LikertDavies (1983) of personal opinion • Availability of resources scale

evaluation of Cronbach’s α 0.43 to • Continuity of care Anchors:health services 0.94 with majority of • Technical quality Stronglyand providers subscales in the 0.70 to • Interpersonal manners disagree

0.79 range • Overall satisfaction StronglyTest-retest was 0.62 to agree0.82

Ware, Synder, Wright, & Davies, 1983). All of the tools used theclient satisfaction dimensions as described by Ware et al. (1978).Nine of the tools measured four or more of the satisfaction dimen-sions, and four tools incorporated fewer dimensions (Harpole et al.,1996; La Monica et al., 1986; Munro et al., 1994; Risser, 1975). The13 tools were further assessed for their appropriateness in a CHCsetting, including their relevance for a PHC model and acceptabil-ity to the population.

Overall, review of the selected tools indicated that clients were notusually involved in developing client satisfaction measurement strat-egies. Four tools were an exception. These tools (McKinley et al.,


1997; Megivern et al., 1992; Risser, 1975; Ware et al., 1983) usedterminology that clients use, and clients were the major source forthe items. In a CHC that uses a PHC service delivery model in whichcommunity participation is a guiding principle, it is important thatclient satisfaction evaluation reflect clients’ perspective on satisfac-tion (Mahon, 1996)

Many of the tools investigated did not have multidisciplinary pro-vider input (a core component of PHC), but often measured satisfac-tion specifically with nursing care or referred to nurses as providers(Eriksen, 1995; La Monica et al., 1986; Risser, 1975). Marsh (1999)claimed to measure client satisfaction outcomes across provider dis-ciplines. However, on closer inspection the only disciplines truly beingmeasured in the tool were nurses and physicians (Marsh, 1999).These tools were subsequently reviewed with the question, “Couldany provider be substituted for the word ‘nurse’ or ‘doctor’?”

Although most of the tools reviewed measured client satisfaction inan inpatient setting, they were included as they addressed dimen-sions relevant to a CHC setting. Six tools measured a range of ambu-latory care settings: outpatient department, emergency department,out-of-hours care, mental health clinics, community-based residentialcare, and employee assistance programs (Attkisson & Greenfield,1994; Davis & Bush, 1995; Harpole et al., 1996; Mayer, Cates,Mastorovich, & Royalty, 1998; McKinley et al., 1997; Risser, 1975).

Reliability data were provided on 11 of the 13 client satisfactiontools. Test-retest reliability was reported on 2 tools (Attkisson &Greenfield, 1994; Ryan et al., 1995), and alpha reliability (internalconsistency) was consistently reported on 9 tools (Davis & Bush,1995; Eriksen, 1995; Greenfield & Attkisson, 1989; La Monica etal., 1986; Marsh, 1999; McKinley et al., 1997; Munro et al., 1994;Risser, 1995; Ware et al., 1983). One article reported that reliabilitytesting was done but gave no results (Mayer et al., 1998). One tooldid not report any evidence of psychometric testing (Harpole et al.,1996). Several tools with subscales reported alpha reliabilities foreach subscale, indicating that they could possibly be used individu-ally or in different combinations (Attkisson & Greenfield, 1994; Davis& Bush, 1995; Eriksen, 1995; Greenfield & Attkisson, 1989; LaMonica et al., 1986). For example, Ware et al. (1983) had 18 subscalesin the satisfaction tool with a 43-item short form.

Validity testing of the tools was not as evident. Content validity wasreported to have been done on several tools (Attkisson & Greenfield,


1994; Davis & Bush, 1995; La Monica et al., 1986; Megivern et al.,1992; Risser, 1975; Ryan et al., 1995; Ware et al., 1983), and con-struct validity was reported on three others (Davis & Bush, 1995;Marsh, 1999; McKinley et al., 1997). Finally, four of the thirteenarticles mentioned validity testing but reported no values (Mayer etal., 1998; McKinley et al., 1997; Ryan et al., 1995; Ware et al., 1983).The difficulty of establishing construct validity for client satisfac-tion scales is evident in this review, where only three tools discussedconstruct validity.

Use of the Criteria to Establish Priorities among Satisfaction Tools

Following review of the selected client satisfaction tools, two toolswere considered for a more in-depth analysis for use in the CHC.The Emergency Department Patient Satisfaction tool has been usedto measure satisfaction in emergency services (Mayer et al., 1998),and the Service Satisfaction Scale (Attkisson & Greenfield, 1994;Greenfield & Attkisson, 1989) has been used and tested for reliabil-ity and validity in a variety of outpatient settings in primary healthcare clinics that included emergency services. The Service Satisfac-tion Scale was deemed to be a more comprehensive tool that wouldbe appropriate for measuring client satisfaction in a CHC using PHCas the model of service delivery.

Review of the Service Satisfaction Scale

The goal in the development of the Service Satisfaction Scale was toprovide a multidimensional way to assess clients’ responses to healthcare services rendered (Attkisson & Greenfield, 1994). The tool in-cludes four subscales or theoretical domains: personal manner andskill, perceived outcomes, office procedures, and accessibility. Wait-ing time items were also written to cover other areas such as costand experience with emergency services. Item wording was clear,and the item comprehension was judged “perfect” or “good” in previ-ous studies. This scale was found to be highly relevant and appro-priate to heterogeneous groups of clients and had been administeredto evaluate client satisfaction with a diversity of service types(Attkisson & Greenfield, 1994). Data on psychometric performanceof the scale included norms from several different populations, suchas primary health care outpatient clinics, mental health outpatientservices, employee assistance program clients, community-basedresidential care, and Driving Under the Influence (DUI) offenders


(Attkisson & Greenfield, 1994; Greenfield & Attkisson, 1989). TheClient Satisfaction Questionnaire (Larsen, Attkisson, Hargreaves,& Nguyen, 1979), a well-used satisfaction survey, was extensivelycompared to the Service Satisfaction Scale (Attkisson & Greenfield,1994). The Service Satisfaction Scale is a multifactorial scale andtherefore more useful in a PHC setting, whereas the Client Satis-faction Questionnaire measures a unitary general satisfaction fac-tor.

A common problem in satisfaction surveys is lack of variability indegrees of satisfaction. Most client responses “pile up” at the mostsatisfied levels (Attkisson & Greenfield, 1994; Pascoe, 1983; Williamset al., 1998). Attkisson and Greenfield (1994) reconfigured responseitems in order to “normalize” the scores and found that the use ofmore extreme endpoint anchors (“delighted” and “terrible”) reducedthe negative skew and ceiling effect.

The recommended approach to administration of the tool is to inviteclients in waiting rooms to participate. A designated research as-sistant (RA), trained in procedures for soliciting voluntary partici-pation of sampled clients, recruits respondents. The RA then leavesthe room when the form is actually being completed. The form isleft in a ballot box when completed. Census samples of all clientsseen at the CHC during a minimum of two typical service weeks isrecommended to ensure that few clients are omitted and that re-spondents are somewhat representative of the target population(Attkisson & Greenfield, 1994). Algorithms are available on requestfrom Greenfield for scoring the subscales of the Service SatisfactionSurvey. We are planning to develop a version that can be opticallyscanned. As well, we are developing procedures to achieve “genera-tion of reports based on the subscales, tailored to and normed forspecific types of settings” (Attkisson & Greenfield, 1994, p. 418). Thesurvey is also an appropriate tool for measuring client satisfactionin each of the specialized service areas within the CHC, as it hasbeen used and tested with various populations in primary healthcare settings (Attkisson & Greenfield, 1994).

Uptake by the CHC Evaluation Team

The criteria developed for tool selection and the results of the clientsatisfaction tool selection and review process were discussed withthe evaluation coordinator and members of the CHC evaluation team.Based on the findings of the project, several options for implemen-


tation were proposed. The most desirable and pragmatic option wasto use the Service Satisfaction Survey tool (Attkisson & Greenfield,1994) without alterations. The tool is psychometrically sound andwas previously used in primary health care settings similar to thoseat the CHC. Three other options were discussed. One was to revisethe Service Satisfaction Survey tool with input from clients who usethe CHC, make some alterations, and then test for reliability andvalidity in a pilot study before using it on the general population.Another option was to use the survey tool and do a comparativeanalysis with a second tool on clients who use the CHC. The lastoption was to use more than one method to gather client satisfac-tion data, such as focus groups comprised of clients using the CHC,and structured interviews involving questions designed by clientsto measure client satisfaction in the CHC.

The evaluation team and the staff of the CHC chose to modify theexisting Service Satisfaction Survey tool (Attkisson & Greenfield,1994) with input from both the clients who use the centre for theirhealth care service and CHC health care providers. Informationabout user friendliness, relevance of questions, and acceptance wasthen obtained from both groups through focus groups. The tool wasmodified based on this input, and a revised tool is being retested forreliability before being used with the CHC clientele. Measures ofinternal consistency through Cronbach’s α scores and test-retestreliability will be obtained. Content validity will be confirmed usingexpert opinion. Factor analysis will determine consistency ofsubscales. The evaluation team also decided to use the revised sat-isfaction survey results in conjunction with focus groups and struc-tured interviews as the preferred method of collecting clientsatisfaction results at the Northeast Community Health Centre.

DISCUSSION

Use of criteria to guide selection of a client satisfaction tool aidedthe evaluation team in choosing a tool that was conceptually andpsychometrically sound. The process helped to eliminate the ardu-ous work of developing a new tool and testing and retesting to en-sure reliability and validity. The criteria assisted in differentiatingbetween psychometrically strong and weak tools and ascertainingsuitability for use in a CHC setting using a PHC model, and poten-tial acceptability to the population using the CHC. However, thecriteria could be more stringent by stating the minimum number ofdimensions that the tool must contain.


Finding a reliable and valid tool from the numerous published toolsavailable is challenging. As there is no “off the shelf” questionnairethat will satisfy the requirements of all service areas (McIver &Meredith, 1998), it is not surprising that health care professionalscontinue to develop their own tools in spite of all the pitfalls. How-ever, the development of tailored instruments for each individualpopulation has some major limitations. There is the inherent diffi-culty in establishing construct validity. Furthermore, there are in-herent sampling biases in most instruments and the difficulty ofnot being able to compare the findings obtained with previous pub-lished results.

If an appropriate tool for measuring client satisfaction can be iden-tified in the literature, the health care provider, the evaluation team,and the community using the CHC must be convinced of its accept-ability and utility. In order to measure client satisfaction in a mean-ingful way for both the CHC staff and the community it serves, itmust be appropriately administered and the results utilized in aresponsive way.

The process for tool selection and modification presented here pro-vides some guidance for those developing similar tools for use withtheir client groups. Developing criteria for comparison of tools basedon appropriateness for a CHC setting is the first step. Applying thosecriteria to potential instruments available in the literature is thenext step. Testing those that fit the criteria with potential clients infocus groups and structured interviews allows for tailoring of theinstrument. The final step is pilot testing the adapted instrumentand making final modifications based on the results.

Are the criteria generated for tool selection and review limited tothis setting, or can they be used in other settings? We suggest thatthe criteria developed for review and selection of a tool for measur-ing client satisfaction can be used in any setting. Can a similar ap-proach be used for other topic areas? The criteria were based ongeneral, minimal requirements, and could provide a useful startingpoint for review of tools measuring another concept.

The criteria we employed to select a potential tool to measure clientsatisfaction in this context were useful. However, some adaptationwas required to incorporate the clients’ perspectives and addresspractical issues specific to the setting. We suggest that, congruentwith the PHC model, a multi-method approach that incorporates fo-


cus groups and exit interviews should be employed in evaluation ofclient satisfaction to add useful information about client perspectives.

ACKNOWLEDGEMENTS

The authors acknowledge the support of Marion Relf, Site Director,Northeast Community Health Centre, Edmonton, Alberta; andWendy Hill, Chief Operating Officer, Northeast Community HealthCentre, Sturgeon Community Hospital and Health Centre, and LeducCommunity Hospital and Health Centre, St. Albert, Alberta. Thisproject was partially funded through a grant provided by the HealthTransition Fund, Health Canada, and supported by Alberta Healthand Wellness.

REFERENCES

Acorn, S., & Barnett, J. (1999). Patient satisfaction: Issues in measure-ment. The Canadian Nurse, 95(6), 33–36.

Atlantic Information Services. (1998/1999). 1998–99 guide to patient satis-faction survey instruments: Profiles of patient satisfaction measure-ment instruments and their use by health plans, employers, hospitals,insurers and physicians (2nd ed.). Washington, DC: Author.

Attkisson, C.A., & Greenfield, T.K. (1994). Client satisfaction questionaire-8 and service satisfaction scale-30. In M. Maruish (Ed.), The use ofpsychological testing for treatment planning and outcome assessment(pp. 402–420). Hillsdale, NJ: Lawrence Erlbaum.

Avis, M., Bond, M., & Arthur, A. (1997). Questioning patient satisfaction:An empirical investigation in two outpatient clinics. Social Scienceand Medicine, 44, 85–92.

Bear, M., & Bowers, C. (1998). Using a nursing framework to measureclient satisfaction at a nurse-managed clinic. Public Health Nurs-ing, 15(10), 50–59.

Burns, N., & Grove, S. (1993). The practice of nursing research: Conduct,critique & utilization (2nd ed.). Toronto: W.B. Saunders.

Church, J., & Lawrence, S. (1999). Community health centres: Innovationin health management and delivery. In J. Hibberd & D.L. Smith(Eds.), Nursing management in Canada (pp. 219–235). Toronto: W.B.Saunders.


Comley, A., & Beard, M. (1998). Toward a derived theory of patient satis-faction. Journal of Theory Construction & Testing, 2(2), 44–50.

Davis, B., & Bush, H. (1995). Developing effective measurement tools: Acase study of the consumer emergency satisfaction scale. Journal ofNursing Care Quality, 9(2), 26–35.

Davis, D., & Hobbs, G. (1989, June). Measuring outpatient satisfactionwith rehabilitation services. Quality Review Bulletin, 192–197.

Eriksen, L. (1995). Patient satisfaction with nursing care: Concept clarifi-cation. Journal of Nursing Measurement, 3(1), 59–76.

Forbes, D. (1996). Clarification of the constructs of satisfaction and dissat-isfaction with home care. Public Health Nursing, 13(6), 377–385.

Forbes, D., & Neufeld, A. (1997). Strategies to address the methodologicalchallenges of client-satisfaction research in home care. CanadianJournal of Nursing Research, 29(2), 69–77.

Graveley, E., & Littlefield, J. (1992). A cost-effectiveness analysis of threestaffing models for the delivery of low-risk prenatal care. AmericanJournal of Public Health, 82(2), 180–184.

Greenfield, T., & Attkisson, C. (1989). Steps toward a multifactorial satis-faction scale for primary care and mental health services. Evalua-tion and Program Planning, 12, 271–278.

Haas, M. (1999). A critique of patient satisfaction. Health InformationManagement, 12, 9–13.

Harpole, L., Orav, J., Hickey, M., Posther, K., & Brennan, T. (1996). Pa-tient satisfaction in the ambulatory setting: Influence of data col-lection methods and sociodemographic factors. Journal of GeneralInternal Medicine, 11, 431–434.

Hill, J. (1997). Patient satisfaction in a nurse-led rheumatology clinic. Jour-nal of Advanced Nursing, 25, 347–354.

La Monica, E., Oberst, M., Madea, A., & Wolf, R. (1986). Development of apatient satisfaction scale. Research in Nursing & Health, 9, 43–50.

Larsen, D.L., Attkisson, C.C., Hargreaves, W.A., & Nguyen, T.D. (1979).Assessment of client/patient satisfaction: Development of a generalscale. Evaluation & Program Planning, 2, 197–207.


Larsson, G., Larsson, B., & Munck, I. (1998). Refinement of the question-naire “quality of care from the patient’s perspective” using struc-tural equation modeling. Scandinavian Journal of Caring Sciences,12, 111–118.

Linder-Pelz, S. (1982). Toward a theory of patient satisfaction. Social Sci-ence & Medicine, 16, 577–582.

Mahon, P. (1996). An analysis of the concept “patient satisfaction” as itrelates to contemporary nursing care. Journal of Advanced Nurs-ing, 24, 1241–1248.

Marsh, G. (1999). Measuring patient satisfaction outcomes across providerdisciplines. Journal of Nursing Measurement, 7(1), 47–62.

Mayer, T., Cates, R., Mastorovich, M., & Royalty, D. (1998). Emergencydepartment patient satisfaction: Customer service training improvespatient satisfaction and ratings of physician and nurse skill. Jour-nal of Healthcare Management, 43(5), 427–440.

McGee, J., Goldfield, N., Riley, K., & Morton, J. (1997). Collecting infor-mation from health care consumers: A resource manual of tested ques-tionnaires and practical advice. Gaithersburg, MD: Aspen.

McIver, S., & Meredith, P. (1998). There for the asking: Can the govern-ment’s planned annual survey really measure patient satisfaction?Health Service Journal, 19, 26–27.

McKinley, R., Manku-Scott, T., Hastings, A., French, D., & Baker, R. (1997).Reliability and validity of a new measure of patient satisfaction without-of-hours primary medical care in the United Kingdom: Develop-ment of a patient questionnaire. British Medical Journal, 314, 193–198.

Megivern, K., Halm, M., & Jones, G. (1992). Measuring patient satisfac-tion as an outcome of nursing care. Journal of Nursing Care Qual-ity, 6(4), 9–24.

Munro, B., Jacobsen, B., & Brooten, D. (1994). Re-examination of the psy-chometric characteristics of the La Monica–Oberst patient satisfac-tion scale. Research in Nursing & Health, 17, 119–125.

Pascoe, G. (1983). Patient satisfaction in primary health care: A literaturereview and analysis. Evaluation & Program Planning, 6, 185–210.


Risser, N. (1975). Development of an instrument to measure patient satis-faction with nurses and nursing care in primary care settings. Nurs-ing Research, 24(19), 45–52.

Ryan, M., Collins, F., Dowd, J., & Pierce, P. (1995). Measuring patientsatisfaction: A case study. Journal of Nursing Care Quality, 9(2),44–53.

Steiber, S., & Krowinski, W. (1990). Measuring and managing patient sat-isfaction. Chicago: American Hospital Publishing.

Strasser, S., & Davis, R. (1991). Measuring patient satisfaction for improvedpatient services. Management Series. Ann Arbor, MI: American Col-lege of Health Care Executives.

Ware, J., Jr. (1981). How to survey patient satisfaction. Drug Intelligence& Clinical Pharmacy, 15, 892–899.

Ware, J., Jr., & Davies, A. (1983). Behavioral consequences of consumerdissatisfaction with medical care. Evaluation & Program Planning,6, 291–297.

Ware, J., Jr., Davies-Avery, A., & Stewart, A. (1978). The measurementand meaning of patient satisfaction: A review of the literature. Health& Medical Services Review, 1, 1–15.

Ware, J., Jr., Snyder, M., Wright, R., & Davies, A. (1983). Defining andmeasuring patient satisfaction with medical care. Evaluation & Pro-gram Planning, 6, 247–263.

Williams, B. (1994). Patient satisfaction: A valid concept? Social Science& Medicine, 38(4), 509–516.

Williams, B., Coyle, J., & Healy, D. (1998). The meaning of patient satis-faction: An explanation of high reported levels. Social Science &Medicine, 47(9), 1351–1359.

Date post:	20-Jul-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

EVALUATION OF CLIENT SATISFACTION IN A COMMUNITY … · la participation communautaire lors de...

Documents