Dental Pilot Project Rules Advisory Committee Monday, October 28, 2019 9:00 AM – 11:00 AM Location OHA Public Health Division 800 NE Oregon Street Portland, OR 97232 Room 900 – Ninth Floor Conference Call Option Dial-In Number: 1-888-273-3658 Participant Code: 766409 Time Agenda 9:00 am – 9:10 am Introductions & Housekeeping 9:10 am – 9:30 am Review of the Rulemaking Process Review of Statement of Fiscal Impact 9:30 am – 10:45 am Review Draft Amended Rules 10:45 am – 10:55 am Public Comment 10:55 am – 11:00 am Next Steps 1
Transcript
Dental Pilot Project Rules Advisory Committee Monday, October 28, 2019
9:00 AM – 11:00 AM
Location OHA Public Health Division 800 NE Oregon Street Portland, OR 97232 Room 900 – Ninth Floor
(D) Documentation of the funding sources for the project;
(E) Documentation of liability insurance relevant to services provided by trainees; and
(F) A statement of previous experience in providing related health care services.
(c) Instructor and Supervisor information:
(A) The criteria used to select instructors and supervisors;
(B) Instructor-to-trainee ratio;
(C) The background of instructors in training techniques and methodology;
(D) The number of proposed supervisors and qualification of supervisors; and
(E) An explanation of how instructors and supervisors will be oriented to their roles and
responsibilities and these rules.
(d) A training program that includes, but is not limited to, a description of:
(A) The instructional content required to meet the level of competence;
(B) The skills trainees are to learn;
(C) The methodology utilized in the didactic and clinical phases;
(D) The evaluation process used to determine when trainees have achieved the level of
competence;
(E) The amount of time required to complete the didactic and clinical phases; and
(F) The level of competence the trainee shall have before entering the employment/utilization
phase of the project.
(e) TraineesTrainee Information:
(A) The criteria that will be used to select trainees;
(B) The number of proposed trainees;
(C) The proposed scope of practice for trainees; and
(D) Information regarding the background check process for participants to determine
compliance with OAR 333-010-0760, Minimum Standards.
(gf) Employment/utilization sites:
(A)site information: A list of all employment/utilization sites locations or class of locations the
proposed project intends to for use; and during the employment/utilization phase where a trainee
may provide care.
(B) Documentation that shows that each site listed meets the definition of an
employment/utilization site.
(g) Underserved Population Information:
(A) A list of the underserved populations the project intends to serve.
(B) Documentation demonstrating that shows the populations the pilot projects intends to serve
are underserved populations;
(C) Documentation demonstrating that each site listed meets the definition of an of the project’s
trainees or employment/utilization sites shall provide services to the underserved populations
identified in the application at a rate of at least 75% 51% of all individuals served by the trainee
or employment/utilization site on a quarterly basis.
(h) CostsCost Information:
(A) The average cost of preparing a trainee, including but not limited to the costs related to
instruction, instructional materials and equipment, space for conducting didactic and clinical
phases, and other pertinent costs;
(B) The estimated cost of care provided in the project; the likely cost of this care if performed by
the trainees of the project; and the cost for provision of this care by current providers.
9
RAC – Dental Pilot Project Program – 10.28.2019
Page 5 of 15
(C) A budget narrative that lists costs associated with key project areas, including but not limited
to:
(i) Personnel and fringe benefits for project director, project dental director, instructors, and staff
associated with the project;
(ii) Contractors and consultants to the project;
(iii) Materials and supplies used in the clinical, didactic, and employment/utilization phases of
the project;
(iv) Equipment and other capital costs associated with the project; and
(v) Travel required for implementing and monitoring the project.
(i) An explanation of the feasibility of achieving the project objectives.
(j) A preliminary evaluation plan that includes, but is not limited to:
(A) How the project sponsor will monitor and evaluate the project;, including but not limited to:
(i) how the project sponsor will monitor and evaluate the rate of underserved populations served
by the pilot project’s trainees or employment/utilization sites; and
(ii) how the project sponsor will monitor and evaluate to ensure trainees are adequately
supervised. Supervision must ensure patients receive quality care and patient health is protected.
(B) A description of the key project activities and their intended effects;
(C) How the project sponsor intends to use the evaluation results for program improvement and
decision making; and
(D) A description of intended patient how the project will measure its progress toward meeting
the goals listed in the application, as described in subsection (4)(a) of this rule. The project must
track and identify measurable project outcomes and metrics as outlined in the requirements under
OAR 333-010-0780, Pilot Project Evaluation and Monitoring by Sponsor.
(k) An identified clinical evaluator who will conduct the clinical evaluation of the project in
accordance with the evaluation plan.
(5) The application must demonstrate that each of the project’s trainees or
employment/utilization sites shall provide services to the underserved populations identified in
the application at a rate of at least 75% of all individuals served by the trainee or employment
utilization site on a quarterly basis.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0730
Dental Pilot Projects: Application Review Process
(1) The Authority shall review an application to determine if it is complete within 60 calendar
days from the date the application was received.
(a) If an applicant does not provide all the information required, and the application is considered
incomplete, then the Authority shall notify the applicant of the information that is missing and
shall allow the applicant 30 calendar days to submit the missing information.
(b) If an applicant does not submit the missing information within the timeframe specified in the
notice, then the application shall be rejected as incomplete. An applicant whose application is
rejected as incomplete may reapply at any time.
(2) An application deemed complete will continue through a review process.
(3) The Authority may have individuals outside the Authority, including representatives of
appropriate professional societies and licensing boards, review applications, but no individual
10
RAC – Dental Pilot Project Program – 10.28.2019
Page 6 of 15
who has contributed to or helped prepare an application will be permitted to conduct a review of
that application.
(4) The Authority may request additional information from an applicant during the review
process.
(5) Once the Authority completes an application review, a Notice of Intent to provisionally
approve or deny an application will be provided to the applicant. The Notice will be sent to
interested parties and will be posted for public comment for a period of 30 calendar days, along
with a link to the application and other materials submitted by the applicant.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0740
Dental Pilot Projects: Project Application Provisional Approval or Denial
(1) Following the close of the public comment period described in OAR 333-010-0730,
Application Review Process, the Authority shall review the public comments that were received
and issue within 30 calendar days of the close of the public comment period:
(a) A provisional decision to grant approval of an application; or
(b) A denial of the application.
(2) If the application is provisionally approved, the project sponsor must comply with the
requirements in OAR 333-010-0750, Provisional Approval; Final Approval, before it can receive
final approval. Projects that receive provisional approval may begin to provide didactic training
however they may not operate or treat live patients until final approval is received from the
Authority.
(3) If the Authority denies the application, the denial must be in writing and must describe the
reasons for the denial. An application may be denied for any of these reasons:
(a) The application does not demonstrate that the project can will meet the minimum standards or
other provisions in these rules;
(b) The application does not demonstrate each of the project’s trainees or employment/utilization
sites shall provide services to the underserved populations identified in the application at a rate
of at least 75% 51% of the individuals served by the trainee or employment/utilization site on a
quarterly basis;
(c) The application does not demonstrate that the project is financially feasible; or
(cd) The Authority has previously approved a similar project.
(4) A sponsor whose project has been denied may not submit a new application within six
months from the date the Authority denied the application.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0750
Dental Pilot Projects: Provisional Approval; Final Approval
(1) A project sponsor that has been provisionally approved must, within 90 calendar days of
provisional project approval, submit the following to the Authority for approval:
(a) A detailed evaluation and monitoring plan that meets the requirements in OAR 333-010-
0780, Pilot Project Evaluation and Monitoring by Sponsor.
11
RAC – Dental Pilot Project Program – 10.28.2019
Page 7 of 15
(b) Written standard operating policies and procedures for the project that ensure compliance
with OAR 333-010-0760, Minimum Standards. Standard operating policies and procedures shall
include, but are not limited to:
(A) Clinical policies and procedures that describe the steps required for implementation of the
project at each site;
(B) Administrative policies and procedures that describe protocols;
(C) Administrative protocols for mandatory record keeping;
(D) Data collection policies and procedure protocols that:
(i) Require data capture and data entry, including identification of the staff positions or other
individuals responsible for these activities;
(ii) Define policies for protection and security of patient data;
(E) The protocol for orientating supervisors to their roles and responsibilities; and
(F) The process for ensuring that potential problems and root causes for deviations and non-
conformances are identified, possible consequences assessed, actions to prevent recurrence
considered, and corrective actions are taken if necessary.
(2) The Authority will review the documentation required in section (1) of this rule and notify
the project sponsor if the plan and policies and procedures are acceptable. The Authority may
request additional information and may request that the project sponsor revise the plan or
policies and procedures to meet the requirements in these rules.
(3) Once the Authority has received an acceptable plan and policies and procedures, it will notify
the project sponsor that the project has been approved along with the plan and policies and
procedures. The final approval letter shall include:
(a) The permitted scope of the project;
(b) Any conditions the Authority deems are necessary to protect patient safety; or ensure quality
of care;
(c) Procedures for which the project will be required to obtain written informed consent for
treatment under OAR 333-010-0770, Informed Consent; and
(d) The length of time the project can operate - from between three to five years.
(4) The Authority shall notify the Oregon Board of Dentistry when a project is approved.
(5) The Authority may deny an application if:
(a) the project fails to timely submit the documents described in subsection (1) that satisfy these
rules;
(b) the project fails to submit additional information or revised plans, policies, or procedures that
are acceptable to the Authority as required by subsection (2) of this rule; or
(c) the documentation submitted by the project under this rule fails to demonstrate that the
project will meet the minimum standards or other provisions in these rules.
(6) A denial issued under this rule must be in writing and must describe the reasons for the
denial.
(7) A sponsor whose project has been denied may not submit a new application within six
months from the date the Authority denied the application.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0760
Dental Pilot Projects: Minimum Standards
An approved dental pilot project shall must:
12
RAC – Dental Pilot Project Program – 10.28.2019
Page 8 of 15
(1) Provide for Ensure patient safety and quality of care as follows:
(a) Comply with informed consent in accordance with OAR 333-010-0770, Informed Consent;
(b) Prohibit a trainee from performing procedures the trainee is not capable of performing based
on the trainee’s level of education, training and experience, physical or mental disability, or
which are outside of the trainee’s approved scope of practice as outlined in the approved
application by the Authority;
(c) Provide or arrange for emergency treatment for a patient currently receiving treatment and
needs emergency care;
(d) Not use the behavior management technique of Hand Over Mouth (HOM) or Hand Over
Mouth Airway Restriction (HOMAR) on any patient;
(e) Comply with ORS 419B.005 to 419B.010 related to the mandatory reporting of child abuse;
(f) Comply with ORS 453.605 to 453.755 or rules adopted pursuant thereto relating to the use of
X-ray machines;
(g) Comply with ORS 679.520 or rules adopted pursuant thereto relating to the treatment of
dental waste materials;
(h) Comply with ORS 679.535 or rules adopted pursuant thereto relating to the requirement to
test heat sterilization devices; and
(i) Ensure that project participants involved in direct patient care:
(A) Have not been convicted of any crimes, within the last 10 years, that is a crime of violence or
crime of dishonesty.
(B) Have not been denied or disciplined by a state entity that issues licenses or certificates.
(j) Ensure adequate supervision and evaluation of trainees, including but not limited to:
(A) timely review of trainee procedures and addressing any deficiencies;
(B) monitoring quality of care and addressing any deficiencies;
(C) monitoring for adverse events and addressing any deficiencies;
(D) monitoring and evaluating to ensure trainees provide the minimum standard of care and
addressing any deficiencies.
(2) Ensure that participants in the project, including trainees, do not engage in unprofessional
conduct as that is defined in ORS 676.150.
(3) Ensure that an accurate patient record is prepared and maintained for each person receiving
dental services, regardless of whether any fee is charged. The record shall contain the name of
the trainee rendering the service and include, but is not limited to:
(a) Name and address and, if a minor, name of guardian;
(b) Date and description of examination and diagnosis;
(c) An entry that informed consent has been obtained in accordance with OAR 333-010-0770,
Informed Consent;
(d) Date and description of treatment or services rendered;
(e) Date and description of all radiographs, study models, and periodontal charting;
(f) Health history; and
(g) Date, name of, quantity of, and strength of all drugs dispensed, administered, or prescribed.
(4) Have a sufficient number and distribution of qualified clinical and non-clinical instructors to
meet project objectives, as identified in the approved application.
(5) Provide instruction to trainees following the training program outlined in the approved
application by the Authority.
(6) Assure that trainees achieve a minimal level of competence before they are permitted to enter
the employment/utilization phase. The sponsor must provide notice to the Authority within 14
13
RAC – Dental Pilot Project Program – 10.28.2019
Page 9 of 15
business days of a trainee entering the employment/utilization phase. The notice shall include,
but is not limited to, the following:
(a) Name, work address, electronic mail address and telephone number of the trainee;
(b) Name, work address, electronic mail address, telephone number and license number of the
supervisor;
(c) Information regarding the trainee’s responsibilities and limitations under Oregon Laws 2011,
chapter 716 and these rules; and
(d) A disclaimer that there is no assurance of a future change in law or regulations that will allow
them to practice without a license outside an approved dental pilot project.
(e) Trainee monitoring records shall be provided to the Authority.
(7) Comply with the requirements of the Dental Pilot Projects statute, Oregon Laws 2011,
chapter 716; these rules; and the approved application including, but not limited to, the
evaluation and monitoring plan.
(8) Evaluate quality of care, access, cost, workforce, and efficacy in accordance with the
evaluation and monitoring plan approved by the Authority and as described in OAR 333-010-
0780, Pilot Project Evaluation and Monitoring by Sponsor.
(9) Within 24 hours of any incident involving a patient in the care of a trainee which results in
any medical occurrence that is life-threatening, requires hospitalization, results in disability or
permanent damage, requires medical or surgical intervention or results in death, the sponsor must
ensure that a detailed written report, along with the patient’s complete dental records, is
submitted to the Authority by the supervising dentist.
(10) Submit detailed quarterly monitoring reports in a format prescribed by the Authority that
include but are not limited to the following categories information for the previous quarter:
(a) Accomplishments or highlights.
(b) Challenges faced and continuous quality improvement activities.
(c) Updated project timeline.
(d) Data reports:
(A) A comprehensive breakdown of each of the data points the project is capturing in its
approved evaluation and monitoring plan including anonymized client level data.
(B) Data generated by the clinical evaluator.
(C) Number and type of any adverse event or complication that occurred during the reporting
period.
(D) Underserved population report: Information identifying the percentage of patients served by
each of the pilot project’s trainees or employment/utilization sites that are within the underserved
population identified in the application.
(11) Follow written standard operating policies and procedures approved by the Authority as
outlined in OAR 333-010-0750, Provisional Approval; Final Approval.
(12) Use templates and follow guidelines for the submission of documents and other reporting
requirements as prescribed by the Authority.
(13) Provide care only at Authority approved employment/utilization sites.
(14) Demonstrate that each of the pilot project’s trainees or employment/utilization sites provides
care to the underserved populations identified in the application at a rate of at least 75% 51% of
the total individuals served by the trainee or employment/utilization site on a quarterly basis.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
14
RAC – Dental Pilot Project Program – 10.28.2019
Page 10 of 15
333-010-0770
Dental Pilot Projects: Informed Consent
(1) A sponsor must ensure that each patient or person legally authorized to provide consent on
behalf of the patient:
(a) Is provided written information about the dental pilot project and who will be providing
treatment;
(b) Gives written consent to be treated by the dental pilot project trainee; and
(c) Gives informed consent for treatment by the trainee.
(2) Written information about the project and who will be providing treatment must include, but
is not limited to:
(a) An explanation of the role and status of the trainee, any certification or licenses a trainee may
hold, the education and training of the trainee and the availability of the trainee’s supervisor for
consultation;
(b) An explanation that the patient can refuse care from a trainee without penalty for such a
request; and
(c) A statement that consenting to treatment by a trainee does not constitute assumption of risk
by the patient.
(3) At a minimum, the following language must be included on the document that requests
consent to be treated by the dental pilot project:
"I ____________________ [name of patient or person acting on patient’s behalf] have
received information about this dental pilot project and provider type. I have been given
the opportunity to ask questions and have them fully answered. I have read and
understand the information and I agree to the trainee of this project providing me
treatment."
_____________________________ ___________
Signature of patient or person acting on patient’s behalf Date
(4) Informed consent for treatment:
(a) Each patient must give informed consent to the procedure. Informed consent means the
consent to a procedure obtained by:
(i) Providing a thorough and easily understood explanation to the patient, or patient's guardian,
of the proposed procedures, any available alternative procedures and any risks associated with
the procedures; and
(ii) Asking the patient, or the patient's guardian, if there are any questions and providing
thorough and easily understood answers to all questions asked.
(b) Patient records must document an entry that informed consent for treatment has been
obtained and the date the informed consent was obtained. Documentation may be in the form of
an acronym such as "PARQ" (Procedure, Alternatives, Risks and Questions) or "SOAP"
(Subjective Objective Assessment Plan) or their equivalent;
(c) Informed consent for treatment must be obtained in writing for procedures identified by the
Authority in the application approval letter, and such consent must be included and documented
in the patient’s record; and
(d) A trainee may not perform any procedure for which the patient or patient’s guardian has not
given informed consent provided; however, in the event of an emergency situation, if the patient
is a minor whose guardian is unavailable or the patient is unable to respond, a trainee may render
15
RAC – Dental Pilot Project Program – 10.28.2019
Page 11 of 15
treatment in a reasonable manner according to community standards and in accordance with the
trainees approved scope of practice.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0780
Dental Pilot Projects: Pilot Project Evaluation and Monitoring by Sponsor
A Project Evaluation and Monitoring Plan required under OAR 333-010-0750, Provisional
Approval; Final Approval, must include, but is not limited to:
(1) A logic model to depict the project activities and intended effects;
(2) A description of key evaluation questions to be addressed by the pilot project, including
relevant process and outcome measures;
(3(3) A description of how the project will measure progress towards the goals identified in the
application. Progress measurements must include quantitative metrics.
(4) A detailed description of the baseline data and information to be collected about the
availability or provision of oral health care services, or both, prior to utilization phase;
(45) A detailed description of baseline data and information to be collected about trainee
performance, patient and community satisfaction, and cost effectiveness;
(56) A detailed description of the methodology and data sources to be used in collecting and
analyzing the data about trainee performance, acceptance by patients, quality of care and cost
effectiveness;
(67) Defined measures to evaluate safety and quality of care provided;.
(7(8) A detailed description of how the project sponsor shall ensure compliance with:
(a) All minimum standards in OAR 333-010-0760, including but not limited to adequate
supervision of trainees; and
(b) All terms and conditions of the approved application, including any amendments.
(9) A process for ongoing quarterly monitoring in accordance with OAR 333-010-0760,
Minimum Standards; and
(810) A process for regular evaluation of project activities across the lifecycle of the project for
continuous quality improvement purposes.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0790
Dental Pilot Projects: Authority Responsibilities
(1) Project monitoring. Program staff shall monitor and evaluate approved projects which shall
include, but is not limited to:
(a) Periodically requesting written information from the project to ascertain the progress of the
project in meeting its stated objectives and in complying with program statutes and regulations:
(b) Periodic, but at least annual, site visits to one or more project offices,
employment/utilizations sites, or other locations where trainees are being prepared or utilized;
and
(c) Reviewing the quarterly reports submitted by the project as described in OAR 333-010-0760,
Minimum Standards.
(2) Advisory committee. The Authority may convene an advisory committee for each approved
dental pilot project.
16
RAC – Dental Pilot Project Program – 10.28.2019
Page 12 of 15
(a) Individuals eligible to serve on an advisory committee include but are not limited to:
(A) Representatives from:
(i) The Oregon Board of Dentistry;
(ii) Professional dental organizations or societies;
(iii) Educational institutions;
(iv) Health systems; and
(v) Individuals representing the target population served by the pilot project.
(B) Individuals with an interest in public health, oral health or expanding access to medical and
dental care.
(b) The purpose of the advisory committee is to gather its members’ collective knowledge,
experience, expertise, and insight to assist the Authority in meeting its responsibilities.
(c) If the Authority convenes an advisory committee it will solicit members for an advisory
committee by public announcement; Individuals interested in serving on the committee are
required to complete an application.
(d) From the applications received, the Authority will appoint no more than 15 members who are
willing to undertake the duties of an advisory committee member and adhere to the committee
charter adopted by the Authority. The Authority will notify each applicant in writing whether
they have been appointed to the committee.
(e) An advisory committee member must:
(A) Attend meetings;
(B) Review approved pilot project quarterly reports at the request of the Authority;
(C) Attend approved pilot project site visits if invited; and
(D) Comply with any confidentiality requirements established by the Authority.
(3) Site visits.
(a) Site visits shall include, but are not limited to:
(A) Determination that adequate patient safeguards are being utilized;
(B) Validation that the project is complying with the approved or amended application;
(C) Interviews with project participants and recipients of care; and
(D) Reviews of patient records to monitor for patient safety, quality of care, minimum standard
of care and compliance with the approved or amended application.
(b) If the Authority has convened an advisory committee, representatives of the committee may
be invited by the Authority to participate in the site visit though the Authority may, at its
discretion, limit the number of members who can participate;
(c) Written notification of the date, purpose and principal members of the site visit team shall be
sent to the project director at least 90 calendar days prior to the date of the site visit;
(d) Plans to interview trainees, supervisors, and patients or to review patient records shall be
made in advance through the project director;
(e) An unannounced site visit may be conducted by program staff if program staff have concerns
about patient or trainee safety;
(f) The Authority will provide the project sponsor with at least 14 business days to submit to the
Authority required patient records, data or other documents as required for the site visit; and
(g) Following a site visit the Authority will:
(A) Within 60 calendar days, issue a written preliminary report to the sponsor of findings of the
site visit, any deficiencies that were found, and provide the sponsor with the opportunity to
submit a plan of corrective action;
17
RAC – Dental Pilot Project Program – 10.28.2019
Page 13 of 15
(i) A signed plan of correction must be received by the Authority within 30 calendar days from
the date the preliminary report of findings was provided to the project sponsor;
(ii) The Authority shall determine if the written plan of correction is acceptable no later than 30
calendar days after receipt. If the plan of correction is not acceptable to the Authority, the
Authority shall notify the project sponsor in writing and request that the plan of correction be
modified and resubmitted no later than 10 business days from the date the letter of non-
acceptance was mailed to the project sponsor;
(iii) The project sponsor shall correct all deficiencies within 30 calendar days from the date of
correction provided by the Authority, unless an extension of time is requested from the
Authority. A request for such an extension shall be submitted in writing and must accompany the
plan of correction.
(iv) If the project sponsor does not come into compliance by the date of correction reflected on
the approved plan of correction, the Authority may propose to suspend or terminate the project as
defined under OAR 333-010-0820, Suspension or Termination of Project.
(B) Within 90 calendar days of receipt of a plan of correction, issue a final report to the sponsor;
and
(C) If there are no corrections needed, the Authority will issue a final report within 180 calendar
days.
(4) The Authority may also provide the sponsor with the opportunity to submit a corrective
action plan to address any deficiencies found by the Authority during any project monitoring as
described in subsection (1) of this rule. The Authority shall notify the sponsor in writing of the
requirement to submit a plan of correction. The sponsor must submit and the Authority must
receive the plan of correction by the deadline set in the notification. All of the requirements and
deadlines described in subsection (3) for corrective action plans apply to a project sponsor when
directed to submit a corrective action plan under this subsection (4).
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0800
Dental Pilot Projects: Project Modifications
(1) Any modifications to an approved project shall be submitted in writing to program staff,
except as specified in section (4) of this rule. All modifications require Authority approval.
Modifications include, but are not limited to the following:
(a) Changes in selection criteria for trainees, supervisors, or employment/utilization sites;
(b) Addition of employment/utilization sites; and
(c) Changes in the scope of practice for trainees.
(2) Upon receipt of a request for a modification approval, the Authority will inform the project
sponsor in writing on the timeline for review of the request and decision response deadline.
(3) If the Authority has convened an advisory committee for an approved project, the Authority
may confer with the advisory committee regarding the proposed modification.
(4) Changes in project staff or instructors are not considered a modification and do not require
prior approval by program staff, but shall be reported to the program staff within two weeks after
the change occurs along with the curriculum vitae for the new project staff and instructors.
(5) The Authority may approve or deny a request for modification. A modification may be
denied if:
18
RAC – Dental Pilot Project Program – 10.28.2019
Page 14 of 15
(a) It does not demonstrate that the project can meet the minimum standards or other provisions
in these rules; or
(b) The modification would result in a substantial change to underlying purpose and scope of the
pilot project as originally approved.;
(c) As a result of the modification, the project would no longer demonstrate that each of the
project’s trainees or employment/utilization sites shall provide services to the underserved
populations identified in the application at a rate of at least 75% 51% of the individuals served
by the trainee or employment/utilization site on a quarterly basis; or
(d) The Authority has previously approved a similar project.
(6) Projects are not permitted to implement the proposed modification until approval has been
rendered by the Authority.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0810
Dental Pilot Projects: Discontinuation or Completion of Project
(1) An approved project must notify the Authority in writing if it intends to discontinue its status
as a Dental Pilot Project, at least 60 calendar days prior to discontinuation. Notification must
include a closing report that includes, but is not limited to:
(a) The reasons for discontinuation as a pilot project;
(b) A summary of pilot project activities including the number of persons who entered the
employment/utilization phase; and
(c) A description of the plan to inform trainees of the project’s discontinuation and that they are
precluded from performing the skills authorized under the pilot project after discontinuation
unless the provider type has been legalized by the State of Oregon.
(2) The project must obtain written acknowledgement from trainees regarding notification of the
project’s discontinuation and preclusion from performing skills authorized under the pilot project
after discontinuation, unless the provider type has been legalized and the trainee has met
necessary licensure requirements.
(3) Project completion. A project sponsor must provide a full report of findings to the Authority
within 180 calendar days of the completion of the project in a format prescribed by the
Authority.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
333-010-0820
Dental Pilot Projects: Suspension, Denial or Termination of Project
(1) A pilot project may be suspended or, terminated, or denied for violation of 2011 Oregon
Laws, chapter 716 or any of these rules.
(2) Failure of a sponsor or anyone involved with an approved pilot project to cooperate with a
reasonable request for records, interviews or a site visit is grounds for the Authority to suspend
or terminate a project. Failure to cooperate includes, but is not limited to, failure to provide
information or documents in a manner requested by the Authority or within the timeframe
requested by the Authority.
(3) If the Authority determines that a dental pilot project is in violation of 2011 Oregon Laws,
chapter 716 or these rules, the Authority may:
19
RAC – Dental Pilot Project Program – 10.28.2019
Page 15 of 15
(a) Require the sponsor to implement an approved corrective action plan in accordance with
OAR 333-010-0790, Authority Responsibilities; or
(b) Issue a Notice of Proposed Suspension or Notice of Proposed Termination in accordance with
ORS 183.411 through 183.470.
(4) A sponsor who receives a Notice may request an informal meeting with the Authority. A
request for an informal meeting does not toll the period for filing a timely request for a contested
case hearing as described in section (5) of this rule.
(5) If the Authority issues a Notice of Proposed Suspension or Notice of Proposed Termination
the sponsor is entitled to a contested case hearing as provided under ORS chapter 183. The
sponsor has 30 calendar days to request a hearing.
(6) If the Authority terminates a dental pilot project, the order shall specify when, if ever, the
sponsor may reapply for approval of a dental pilot project.
Statutory/Other Authority: 2011 OL Ch. 716
Statutes/Other Implemented: 2011 OL Ch. 716
20
Dental Health Professional Shortage Areas
About Dental HPSA’s: In Oregon, 33 of 36 counties are designated Dental HPSA. A Dental HPSA can be classified as a Dental HPSA in multiple ways. https://bhw.hrsa.gov/shortage-designation/types
• Shortage designations indicate geographic areas (counties or portions of counties) with a shortageof dental providers for a given population – according to the HRSA guidelines.1
Three distinct types of designations are available:2
1. Geographic: the entire population in the designated area is identified as underserved andresources are considered over-utilized.
2. Population: an underserved population identified within a specific area. Eligible populationsinclude:
• Low-income: there must be at least 30% of the population at or below 200% of the Federal PovertyLevel.
• Migrant farmworkers: migrant farmworkers and their non-farm working family members.
• Native Americans: American Indians or Alaska natives that are not part of group that is alreadyautomatically designated.
• Other populations that face access barriers due to language, cultural or disability barriers.
3. Facility: A facility designation is based on the population served and includes comprehensivehealth centers, Federally Qualified Health Centers, rural health centers, correctional facilities,Indian Health Service or tribal health service or other public facility.
Purpose: The purpose of these designations is to identify areas of greatest need in order to prioritize limited resources and target those resources. These designations target millions of dollars of federal resources to improve health care in underserved areas of the state. [OHA Office of Primary Care] estimates these designations bring in over $20 million per year in unmatched federal resources. For example, by being designated as a HPSA means the area is eligible to receive funding under the National Health Service Placement (NHSP) program. Dental students receive full coverage of tuition, fees and other educational costs, plus a monthly stipend to assist with living expenses. For each year of financial support (up to four years), the student agrees to serve one year (minimum two years) at an NHSC-approved site in at an NHSC facility - high-need urban, rural, or frontier community across the nation.3
Key Points: 1. Not all individuals living in Dental HPSA’s are considered underserved.
2. Oregon has two counties, Sherman and Gilliam, where all individuals are considered to haveaccess to dental care challenges as there are a shortage of providers for the entire county.
3. Purpose of HPSA designations are to prioritize limited funding to areas of greatest need.
1 Health Resources and Service Administration, HRSA Health Workforce, Shortage Designation, https://bhw.hrsa.gov/shortage-designation/types 2 Oregon Health Authority, Office of Primary Care, https://www.oregon.gov/oha/HPA/HP-PCO/Pages/index.aspx 3 Health Resources and Service Administration, National Health Services Corps https://nhsc.hrsa.gov/
Subject: RAC-Dental Pilot Project Program - Feedback Submission Process
Attachments: Timeline RAC 2019.pdf
Importance: High
Thank you for attending the RAC meeting this past Monday.
Meeting minutes will be completed and sent to the RAC members by Friday, October 4, 2019.
Based upon the comments and input received on Monday, OHA will hold a second RAC meeting on October 28, 2019 from 9am-11am in the same location, Room 900, 800 NE Oregon Street, Portland, Oregon 97232.
Feedback Submission: RAC members who are interested in submitting feedback to OHA must do so by October 13, 2019.
Instructions for Submission:
1. OHA will only accept feedback from RAC members at this time.2. There will be several opportunities for individuals submit public comments. Public comment will
be accepted at the following times:• At the conclusion of the second RAC meeting on October 28, 2019.• A public comment period will be announced and rulemaking documents will be posted
to our website. Dates TBD.• OHA will hold a public hearing to seek public comments. Date TBD.
Please see the attached timeline for questions.3. Feedback will be accepted via electronic mail, mail or fax. We strongly encourage RAC
members to submit all feedback via electronic mail. Please REPLY ALL to this message sothat all individuals on the RAC receive your feedback.
• Mail: Sarah Kowalski, RDH Dental Pilot Project Coordinator Oral Health Program 800 NE Oregon Street, Suite 875 Portland, Oregon 97232-2186
(Please submit to both email addresses, we request that you include all members of theRAC when submitting feedback.)
Feedback Submission Process - Post RAC Meeting 9.28.2019
23
2
4. All feedback will be shared with members of the RAC if submitted via mail, fax or directly to the email addresses above.
5. All feedback is a matter of public record and will be included in the meeting minutes. Feedback will be consolidated and shared ahead of the next RAC meeting on October 28, 2019 and posted with meeting materials by October 22, 2019.
We sincerely appreciate your attention to these materials and thank you again for taking the time to attend the meeting this past Monday.
As always, if you have questions, please feel free to reach out.
Thank you,
Sarah Kowalski
Sarah Kowalski, RDH, MS
Operations & Policy Analyst 3
Dental Pilot Project Coordinator
Oral Health Program
The Oregon Health Authority
800 NE Oregon Street
Portland, Oregon 97232
971-673-1563 (office)
Website: healthoregon.org/dpp
Feedback Submission Process - Post RAC Meeting 9.28.2019
Feedback Eli Schwarz - RAC - Dental Pilot Project Program
26
Legislative Intent
ODA was part of the very limited, small group of legislatively directed professionally mediated meetings
used to develop consensus on SB 738 in 2011. In fact, two ODA dentists were in the room consistently
through the process, where the other representatives from the Oregon Oral Health Coalition changed
throughout the session. We can speak confidently and accurately to the original intent of those in the
room who came to this consensus on the bill and ultimately to legislative intent when passing the bill.
The original intent of SB 738 was to allow for innovative solutions to be tested in Oregon to expand
access to needed care for those Oregonians with the highest disease rates. The term “focus” was
purposely used to ensure that these creative ideas would prioritize populations that meet two criteria:
1.) are part of a population that evidence-based studies have shown the highest disease rates and 2.)
have the least access to dental care. The legislation uses an “AND” in that sentence, not an “OR.”
Populations being served within the pilot project must meet both criteria- have a high disease rate as
shown by evidence-based studies in addition to having barriers in accessing care. OARs should reflect
both of these criteria.
Additionally, the term “focus” was purposely used to ensure projects concentrated on serving
populations defined above. Focus was presumed and intended to be a plain word choice with its plain
meaning. Our members in the room, who participated in the negotiations, believe this choice of word,
focus, to mean more than a simple majority and to be as many people within the target population as
possible. That is the entire point of the legislation: to serve a population that most needs a creative
solution to care.
Competency and Safety
All current licensed dental providers in Oregon have passed intense board examinations and are
monitored by the Oregon Board of Dentistry to ensure the public of their competency and thus ensuring
public safety. In fact, the primary purpose of the Board of Dentistry is to protect the public. While
providers in dental pilot projects are not subject to licensure through the OBOD, the law does require
that they provide care meeting the same standard of care as those providing similar services with a
license, and requires the OHA to create rules related to “protecting the safety of patients seen or
treated in the project.” We disagree that the usage of the word “ensure” is too strong. Ensuring public
safety and quality of care is clearly within the intent of the law.
Current projects and precedent
As indicated by OHA staff in the last rules meeting, current operating pilot projects are serving their
targeted populations well above the proposed 75% threshold. Pilot Project #100 was said to be serving
about 85-90% of their target population, with Pilot Project #200 being close to 100%. Both of these
projects have shown that you can narrowly tailor a project to ensure trainees are concentrating on
serving the target population. Why would we want to lower the expectation of future projects?
Retroactivity
Feedback Jim McMahan and Conor McNulty - RAC - Dental Pilot Project Program
27
Various RAC members indicated concerns with a retroactive nature of these rules. It would be helpful to
understand what changes within these rules would actually affect current operating projects. On the
surface, it seems that both current pilot projects meet the proposed criteria so no further changes
would be necessary.
If the concern is about a future project that has not yet officially been approved, that project would
have the ability to adjust its program parameters prior to implementation. Hence, no retroactivity there
either. Further clarification of concerns by RAC members would be appreciated on this issue. OHA
should not be limited in rulemaking changes simply because there are existing programs or applications
submitted. This preemption would be completely unique to these programs and OHA undergoes
rulemaking and changes to other programs it operates routinely. The rule making process is designed in
such a way to ensure proper public participation, vetting, and to provide sufficient notice of changes to
those affected.
Suggested changes
• Ideally, raise 75% threshold to 85%. Certainly, do not lower it below 75%.
• Threshold should be for the population being served by the trainee within the project, not the
utilization site. With the inclusion of the utilization site, there is risk that the actual trainee will
not be increasing care to those intended by the authorizing statute.
• Ensure that the focus population being served by a dental pilot project meet both criteria as
defined in the authorizing statute: 1.) being part of a population with high disease rates, AND 2.)
have the least access to dental care.
• In describing patient population, projects should detail the access barriers identified, ensuring
population meets statutory criteria.
We appreciate the opportunity to submit these comments and look forward to working with the
entire Rules Advisory Committee to find language that can be mutually agreed upon, as we did in the
original authorizing legislative mediation.
Sincerely,
James McMahan, DMD
Immediate Past President of the Oregon Dental Association
Conor McNulty, CAE
Executive Director, Oregon Dental Association
Feedback Jim McMahan and Conor McNulty - RAC - Dental Pilot Project Program
28
Rule Advisory Committee
Written comments from Dayna Steringer
for meeting held on 9/30/19
(proposed changes are underlined)
333-010-0700
(3) These rules apply to applications submitted after Feb 1, 2020
Comment: Changes to rules that govern the application process should not apply to
applications or pilots that have already been submitted.
333-010-0710 – Definitions
(25) “Underserved Populations” means …least access to dental care including, but not limited
to: those who receive care in a nationally recognized health professional shortage area.”
Comment: Michigan state law recognizes those who live in a dental shortage area as having a
barrier to accessing dental care. (MCL 333.16654)
333-010-0720
*section 4 is missing letter ‘f’ in the numbering
(4)(h)(C) “…at a rate of at least 51% of all individuals served…by the trainee or
employment/utilization site on an annual basis.”
(5) “…at a rate of at least 51% of all individuals served……by the trainee or
employment/utilization site on an annual basis.”
Comment: retain the “or” as originally suggested by DOJ; change “quarterly” to “annually”
333-010-0730
(3) …but no individual who has contributed to or helped prepare an application will be permitted
to conduct a review of that application.
Comment: this wasn’t a change, but seems to need clarification nonetheless
333-010-0740
Feedback Dayna Steringer - RAC - Dental Pilot Project Program
29
(3)(b) “…at a rate of at least 51% of the individuals served…by the trainee or
employment/utilization site on an annual basis.”
333-010-0760
(14) “…at a rate of at least 51% of the total individuals served…by the trainee or
employment/utilization site on an annual basis.”
Comment: in general, it seems like this would already be required by (7) or (8) of this rule
333-010-0780
(3) “…must include quantitative metrics.”
333-010-0800
(5)(c) “…at a rate of at least 51% of the total individuals served…by the trainee or
employment/utilization site on an annual basis.”
Feedback Dayna Steringer - RAC - Dental Pilot Project Program
30
Colleagues,
We appreciate the opportunity to share our comments with you and hope that you will find
them useful.
Our comments are below:
Underserved Population
As expressed at the meeting on September 30th, we continue to have concerns over the
proposed language in 333-010-0720. We believe that the language as proposed will stifle
innovation and create an unnecessary barrier to future pilot projects.
Senate Bill 738 (2011), established the dental pilot project program to “encourage the
development of innovative practices in oral health care delivery systems with a focus on
providing care to populations that evidence-based studies have shown have the highest disease
rates and the least access to dental care.” Sec. 1 (emphasis added). However, the proposed
revisions take it one step further at OAR § 333-010-0720(4)(h)(C) to require that pilot projects
provide services to the underserved population identified in the pilot application “at a rate of at
least 75% of all individuals served by the trainee or employment utilization site on a quarterly
basis.” See also OAR §§ 333-010-0740(b); 333-010-0760(14), 333-010-0800(5)(c).
Even though current pilot projects most likely meet this standard, we are concerned that this
numeric definition of 75% is an unnecessary and overly-restrictive threshold for any future
projects that are still focusing on underserved populations. Because the application process as
defined by the current rules require a potential project to demonstrate that it would focus on
this population (333-010-0710 (10) and 333-010-0720 (g)(B)), OHA has the discretion to make
that determination based on documentation provided by the applicant. Continuing to allow
OHA this discretion could invite a wider range of pilots and include providers and clinics that
may have a goal of serving more underserved patients through the pilot even if it takes some
time to meet their goals. We are concerned that the requirement to demonstrate a specific
percentage up front will discourage existing oral health providers from participating and thus
expanding their patient base to include more underserved patients, therefore decreasing future
access to care.
Also, by requiring that a certain percentage be met quarterly, there is no room for dipping
below that number in one quarter, while over the course of the pilot the project may be well
above it.
And while the Oregon Dental Pilot Project Program is not specific to dental therapy, there is
evidence in dental therapy legislation from other states that when numeric thresholds are used
in order to help direct services to those who need it most, the highest number that is used is
50% (MN, ME, MI, NV) of dental therapists’ patients and this is one of many other practice
settings with no limitations: FQHCs, Tribal Clinics, School-based health clinics, clinics located in
DHPSAs, home-bound patients, etc…
Finally, the enacting legislation does not require this level of specificity, and intent should not be
assumed where none is documented.
Feedback Christina Peters - RAC - Dental Pilot Project Program
31
Therefore we urge that no percentage-based threshold be used to define focus, and that
current rule language continues to guide selection of projects that will focus on underserved
populations.
If, however, a percentage is used, it should not be more than 50% of the trainees (not entire
clinics) patients, on a yearly average over all practice settings and approved sites.
Supervision, Patient Safety, and Quality of Care
Pilot Project #100 also has concerns about the new proposed language around supervision
requirements and patient care. We have concerns about the following provisions in the
proposed revisions to the OAR:
333-010-0720 regarding applications (note that the section numbering in 0720 is not
accurate). Sections (A)(i) and (ii) at the top of page 5 of 15 add new language to require the
application to explain how the project sponsor will “monitor and evaluate to ensure trainees are
adequately supervised. Supervision must ensure patients receive quality care and patient health
is protected.” (emphasis added).
333-010-0750 regarding the Authority’s provisional and final approval of projects. Section 3(b)
indicates that when the Authority approves a project, it will send a letter to the applicant that
includes “Any conditions the Authority deems are necessary to protect patient safety or ensure
quality of care.” (new language is underlined).
333-010-0760 sets minimum standards for pilot projects and contains two provisions
surrounding supervision and patient care:
o Section 1 previously said that the project must “Provide for patient safety as
follows,” and then lists a number of steps that must be taken. While this
appears to us to be consistent with S.B. 738 (which refers to “provisions for
protecting the safety of patients”), the Authority is now proposing to
change that to instead say “Ensure patient safety and quality of care as
follows.”
o One of the steps under (1) is in new language in a new subsection (j),
requiring that the project “Ensure adequate supervision and evaluation of
trainees, including but not limited to: (A) timely review of trainee
procedures and addressing any deficiencies; (B) monitoring quality of care
and addressing any deficiencies; (C) monitoring for adverse events and
addressing any deficiencies; (D) monitoring and evaluating to ensure
trainees provide the minimum standard of care and addressing any
deficiencies.”
We are concerned that these proposed revisions set up requirements that are not clearly
defined, but are vague and could be highly subjective. For example, we read the language in
0720, requiring that supervision “ensure” patient safety and quality of care, as being a
potentially impossible standard to meet unless it is better defined. We also think it could be
read as being at odds with the language in 0760(1)(j) indicating that the supervision allows for
oversight and the ability to address any deficiencies. Moreover, the purpose of the pilot
Feedback Christina Peters - RAC - Dental Pilot Project Program
32
projects as defined in OAR 333-010-0700 is in part to “evaluate the quality of care.” (emphasis
added).
As currently proposed, we think the revised OARs could make it difficult for a provider to prove
they have “ensured” patient safety and quality of care, particularly when deficiencies may occur
(and are expected to occur and be addressed under 0760(1)(j)), and that is why the OARs have
previously referenced steps to “provide for” patient safety and quality of care. An outright
requirement that safety and quality be guaranteed is a potentially impossible standard to meet
and is not required even under the Oregon Dental Practices Act.
Additionally, as we’ve learned in pilot project 100, “quality of care” and “patient safety” is
subject to opinion and it is impossible to have a standardized definition of either. Our project
has experienced differences in clinical opinion being communicated under the umbrella of
“quality of care” and “patient safety”. In all cases, our patients received quality care and we
have no patient safety issues in our clinics. It is not the purpose of these OARs to subject pilot
project supervisors to vague, unattainable standards.
Pilot projects are required to have certain delineated steps in place for patient safety and quality
of care purposes, along with various defined supervisory requirements and follow-up measures
that can be attained, including:
333-010-0720 (4)(c)(d)
333-010-0760
333-010-0770
333-010-0780 (4-8)
333-010-0790 (1)(3)
333-010-0820
We therefore recommend that the OARs continue to use “provide for” instead of “ensure”
when referencing supervision as tied to patient safety and quality of care.
We appreciate the opportunity to share these comments with you and look forward to the next
meeting. We are happy to answer any questions you may have.
Christina Peters
RAC Pilot Project Committee Member
Tribal Community Health Provider Project Director
Northwest Portland Area Indian Health Board
Pam Johnson
Proxy- RAC Pilot Project Committee Member
Native Dental Therapy Initiative Project Manager
Northwest Portland Area Indian Health Board
Feedback Christina Peters - RAC - Dental Pilot Project Program
33
DRAFT Rulemaking Process
The Oregon Health Authority, Public Health Division has policies and procedures that guide the rulemaking process. In order to have the rules effective February 1, 2020, we will be following the timeline below.
Date Activity
August 2019 Notify previous RAC members of proposed activities.
September 2019 Draft proposed rules and Statement of Need and Fiscal Impact form
September 2019 - October 2019
Convene RAC and hold meetings to seek input on proposed rules and required forms
November 12, 2019 OHA Rules Coordinator needs final proposed rules and rulemaking forms
November 12, 2019 OHA Rules Coordinator will review forms and seek approval to file
November 22, 2019 OHA Rules Coordinator will file the notice of proposed rulemaking with the Oregon Secretary of State
November 27, 2019 Rulemaking documents will be posted to our website and interested parties will be notified
December 1, 2019 Notice appears in the Oregon Bulletin
December 16, 2019 or later Hold public hearings to seek public comments
December 23, 2019 or later Public comment period closes
After Public Comment Period Closes
Respond to comments from the public comment period
January 24, 2020
Final rule text showing changes and responses to public comment period due to the OHA Rules Coordinator
OHA Rules Coordinator will file the final rules with the Oregon Secretary of State
February 1, 2020 Rules are effective upon filing
34
CENTER FOR PREVENTION AND HEALTH PROMOTIONOral Health Program
Kate Brown, Governor
Dental Pilot Project Program Rules Advisory Committee Meeting Minutes
Date: Monday, September 30, 2019Time: 9:00 AM – 11:00 PMLocation: OHA Public Health Division
800 NE Oregon StreetPortland, OR 97232Conference Room 900- 9th Floor
800 NE Oregon St, Ste 825Portland, Oregon 97232-2186
Agenda Item: Review items noted in red throughout Rules Draft; Review new, modified and/or deleted language under 333-010-0700
Meeting minutes include portions of the administrative rules OAR 333-010-0700 through 333-010-0820. A full copy of the OARs are available in the meeting materials.
333-010-0700
Dental Pilot Projects: Purpose (1) The Dental Pilot Projects are intended to evaluate the quality of care, access, cost, workforce,
and efficacy by teaching new skills to existing categories of dental personnel; developing new
categories of dental personnel; accelerating the training of existing categories of dental
personnel; or teaching new oral health care roles to previously untrained persons. The purpose of
Dental Pilot Projects are to encourage the development of innovative practices in oral health care
delivery systems with a focus on providing care to underserved populations that evidence-based
studies have shown have the highest disease rates and the least access to dental care.(2) These
rules establish the requirements of Dental Pilot Project applications; the process for reviewing
applications; approval or denial of applications; minimum standards for approved projects;
evaluation and monitoring of Dental Pilot Projects; suspension or termination of an approved
Dental Pilot Project; and discontinuation or closure of a project.
(3) These rules apply to all approved dental pilot projects and any dental pilot project applicants,
including any applicants with pending applications.
(a) Applications for Dental Pilot Projects received on or after December 1, 2018; and
(b) Dental Pilot Projects approved before or after December 1, 2018.
(4) A dental pilot project that was approved and was operating before December 1, 2018 has
until June 1, 2019 to come into compliance with the minimum standards in OAR 333-010-0760.
Comments: • Dayna Steringer: concern made known regarding applying rules
retroactively
• Molly Johnson: similar concern of retroactively applying rules to
projects already underway
Review new, modified and/or deleted language under 333-010-0710
333-010-0710
Dental Pilot Projects: Definitions For purposes of OAR 333-010-0700 through 333-010-0820, the following definitions apply:
(1) "Adverse event" means unnecessary harm caused by due to dental treatment., regardless of
whether it is associated with error or considered preventable.
(2) "Authority" means the Oregon Health Authority.
(3) "Business day" means any 24-hour day other than a Saturday, Sunday or federal or state legal
holiday.
(4) "Clinical evaluator" means a dentist, licensed in the State of Oregon or another state, who is
responsible for conducting an independent clinical evaluation of an approved dental pilot project;
who is unaffiliated with the project; and who has no financial or commercial interest in the
Dayna Steringer: request copy of journals which define adverse event Note: Please see reference materials at the end of the minutes.
333-010-0710
Dental Pilot Projects: Definitions (10) "Employment/utilization site" means an Authority approved site for use during the
employment/utilization phase that provides care to underserved populations. that evidence has
shown have the highest disease rates and the least access to dental care. An employment
utilization site includes any location where dental health care services are provided by a project’s
trainees.
(25) “ Underserved Populations” means groups of individuals that evidence-based studies have
shown have the highest disease rates and the least access to dental care including, but not limited
to: low-income, rural populations, or uninsured populations.
Comments:
Dayna Steringer: Senate Bill does not include “underserved population”
Review new, modified and/or deleted language under 333-010-0720
333-010-0720
Dental Pilot Projects: Application Procedure (g) Employment/utilization sites information:
(A) A list of all employment/utilization sites the proposed project intends to use; and (B) Documentation that shows that each site listed meets the definition of an employment/utilization site. (h) Underserved Population Information:
(A) A list of the underserved populations the project intends to serve.
(B) Documentation demonstrating that the populations the pilot project intends to serve are
underserved populations;
(C) Documentation demonstrating that each of the project’s trainees or employment/utilization
sites shall provide services to the underserved populations identified in the application at a rate
of at least 75% of all individuals served by the trainee or employment utilization site on a
quarterly basis.
Comments:
Pam Johnson: A and B seem duplicative. Provide list of population and documentation showing they are underserved.
Conor McNulty: May be helpful to note list of barriers/high needs.
Jim McMahon: Does “or employment utilization site” mean they might not see underserved population. Think it should be “and”
Laura McKeane: Why 75%?
Sarah Kowalski: Refer back to meaning of focus and turning it into a quantitative
Review new, modified and/or deleted language under 333-010-0740
333-010-0740
Dental Pilot Projects: Project Application Provisional Approval or Denial
(1) Following the close of the public comment period described in OAR 333-010-0730,
Application Review Process, the Authority shall review the public comments that were received
and issue within 30 calendar days of the close of the public comment period:
(a) A provisional decision to grant approval of an application; or
(b) A denial of the application.
(2) If the application is provisionally approved, the project sponsor must comply with the
requirements in OAR 333-010-0750, Provisional Approval; Final Approval, before it can receive
final approval. Projects that receive provisional approval may begin to provide didactic training
however they may not operate or treat live patients until final approval is received from the
Authority.
(3) If the Authority denies the application, the denial must be in writing and must describe the
reasons for the denial. An application may be denied for any of these reasons:
(a) The application does not demonstrate that the project will can meet the minimum standards or
other provisions in these rules;
(b) The application does not demonstrate each of the project’s trainees or employment/utilization
sites shall provide services to the underserved populations identified in the application at a rate
of at least 75% of the individuals served by the trainee or employment utilization site on a
quarterly basis;
(cb) The application does not demonstrate that the project is financially feasible; or
(dc) The Authority has previously approved a similar project.
(4) A sponsor whose project has been denied may not submit a new application within six
months from the date the Authority denied the application.
Comments:
Pam Johnson: seems redundant; raises question why this one is being highlighted
Review new, modified and/or deleted language under 333-010-0750 333-010-0750
Dental Pilot Projects: Provisional Approval; Final Approval (3) Once the Authority has received an acceptable plan and policies and procedures, it will notify
the project sponsor that the project has been approved along with the plan and policies and
procedures. The final approval letter shall include:
(a) The permitted scope of the project;
(b) Any conditions the Authority deems are necessary to protect patient safety or ensure quality
of care;
(c) Procedures for which the project will be required to obtain written informed consent for
treatment under OAR 333-010-0770, Informed Consent; and
39
(d) The length of time the project can operate - from between three to five years.
(4) The Authority shall notify the Oregon Board of Dentistry when a project is approved.
(7) A sponsor whose project has been denied may not submit a new application within six
months from the date the Authority denied the application.
Comments:
Jim McMahon: should be “and” between patient safety and quality of care 333-010-0750
Dental Pilot Projects: Provisional Approval; Final Approval (5) The Authority may deny an application if:
(a) the project fails to timely submit the documents described in subsection (1) that satisfy these
rules;
(b) the project fails to submit additional information or revised plans, policies, or procedures that
are acceptable to the Authority as required by subsection (2) of this rule; or
(c) the documentation submitted by the project under this rule fails to demonstrate that the
project will meet the minimum standards or other provisions in these rules.
(6) A denial issued under this rule must be in writing and must describe the reasons for the
denial. Comments:
Dayna Steringer: “c” seems duplicative
Erin Williams: includes all documentation from project; wanted to include for provisional and final denial; there are additional documents to be submitted
Sarah Kowalski: Evaluation Plan is separate, approved after project receives provisional approval.
Review new, modified and/or deleted language under 333-010-0760 333-010-0760
Dental Pilot Projects: Minimum Standards (A) Have not been convicted of any crimes, within the last 10 years, that is a crime of violence or
crime of dishonesty.
(B) Have not been denied or disciplined by a state entity that issues licenses or certificates.
(j) Ensure adequate supervision and evaluation of trainees, including but not limited to:
(A) timely review of trainee procedures and addressing any deficiencies;
(B) monitoring quality of care and addressing any deficiencies;
(C) monitoring for adverse events and addressing any deficiencies;
(D) monitoring and evaluating to ensure trainees provide the minimum standard of care and
addressing any deficiencies. Comments:
Erin Williams: needed to put this in to account what is already happening since it is not currently there; listing out what should be in the plan around supervision
40
333-010-0760
Dental Pilot Projects: Minimum Standards (d) Data reports:
(A) A comprehensive breakdown of each of the data points the project is capturing in its
approved evaluation and monitoring plan including anonymized client level data.
(B) Data generated by the clinical evaluator.
(C) Number and type of any adverse event or complication that occurred during the reporting
period.
(e) Underserved population report: Information identifying the percentage of patients served by
each of the pilot project’s trainees or employment/utilization sites that are within the underserved
population identified in the application.
(11) Follow written standard operating policies and procedures approved by the Authority as
outlined in OAR 333-010-0750, Provisional Approval; Final Approval.
(12) Use templates and follow guidelines for the submission of documents and other reporting
requirements as prescribed by the Authority.
(13) Provide care only at Authority approved employment/utilization sites.
(14) Demonstrate that each of the pilot project’s trainees or employment utilizations sites
provides care to the underserved populations identified in the application at a rate of at least 75%
of the total individuals served by the trainee or employment utilization site on a quarterly basis.
Comments:
Pam Johnson: sounds like a separate document; is this a new requirement?
Sarah Kowalski: just information reporting out that your project is already doing
Erin Williams: would it make more sense to fold new “e” into data reports as a separate bullet?
Jim McMahon: would like to go deeper and include more demographic data on population
Review new, modified and/or deleted language under 333-010-0780 333-010-0780
Dental Pilot Projects: Pilot Project Evaluation and Monitoring by Sponsor
A Project Evaluation and Monitoring Plan required under OAR 333-010-0750, Provisional
Approval; Final Approval, must include, but is not limited to:
(1) A logic model to depict the project activities and intended effects;
(2) A description of key evaluation questions to be addressed by the pilot project, including
relevant process and outcome measures;
(3) A description of how the project will measure progress towards the goals identified in the
application. Progress must be measured with quantitative metrics.
(34) A detailed description of the baseline data and information to be collected about the
availability or provision of oral health care services, or both, prior to utilization phase;
(45)A detailed description of baseline data and information to be collected about trainee
performance, patient and community satisfaction, and cost effectiveness;
41
(56) A detailed description of the methodology and data sources to be used in collecting and
analyzing the data about trainee performance, acceptance by patients, quality of care and cost
effectiveness;
(67) Defined measures to evaluate safety and quality of care provided.
(8) A detailed description of how the project sponsor shall ensure compliance with;
(a) All minimum standards in OAR 333-010-0760, including but not limited to adequate
supervision of trainees; and
(b) All terms and conditions of the approved application, including any amendments.
(97) A process for ongoing quarterly monitoring in accordance with OAR 333-010-0760,
Minimum Standards; and
(108) A process for regular evaluation of project activities across the lifecycle of the project for
continuous quality improvement purposes.
Comments:
Pam Johnson: should say “must include quantitative metrics” versus “must be measured with…”
Review new, modified and/or deleted language under 333-010-0790 333-010-0790
Dental Pilot Projects: Authority Responsibilities (4) The Authority may also provide the sponsor with the opportunity to submit a corrective
action plan to address any deficiencies found by the Authority during any project monitoring as
described in subsection (1) of this rule. The Authority shall notify the sponsor in writing of the
requirement to submit a plan of correction. The sponsor must submit and the Authority must
receive the plan of correction by the deadline set in the notification. All of the requirements and
deadlines described in subsection (3) for corrective action plans apply to a project sponsor when
directed to submit a corrective action plan under this subsection (4).
Comments: Erin Williams: the suggestion to add this is to add option for a corrective plan after reporting or after site visits
Laura McKeane: is there a place where it defines timeline on notices?
Erin Williams: if in writing, it would be 30 days (see i on page 12) Review new, modified and/or deleted language under 333-010-0800 333-010-0800
Dental Pilot Projects: Project Modifications (1) Any modifications to an approved project shall be submitted in writing to program staff,
except as specified in section (4) of this rule. All modifications require Authority approval.
Modifications include, but are not limited to the following:
42
(a) Changes in selection criteria for trainees, supervisors, or employment/utilization sites;
(b) Addition of employment/utilization sites; and
(c) Changes in the scope of practice for trainees.
(2) Upon receipt of a request for a modification approval, the Authority will inform the project
sponsor in writing on the timeline for review of the request and decision response deadline.
(3) If the Authority has convened an advisory committee for an approved project, the Authority
may confer with the advisory committee regarding the proposed modification.
(4) Changes in project staff or instructors are not considered a modification and do not require
prior approval by program staff, but shall be reported to the program staff within two weeks after
the change occurs along with the curriculum vitae for the new project staff and instructors.
(5) The Authority may approve or deny a request for modification. A modification may be
denied if:
(a) It does not demonstrate that the project can meet the minimum standards or other provisions
in these rules; or
(b) The modification would result in a substantial change to underlying purpose and scope of the
pilot project as originally approved.
(c) As a result of the modification, the project would no longer demonstrate that each of the
project’s trainees or employment/utilization sites shall provide services to the underserved
populations identified in the application at a rate of at least 75% of the individuals served by the
trainee or employment utilization site on a quarterly basis; or
(d) The Authority has previously approved a similar project.
(6) Projects are not permitted to implement the proposed modification until approval has been
rendered by the Authority.
Comments: Jim McMahon: same comment regarding “and/or”
Pam Johnson: “c” is unnecessary; a captures it fully
Dayna Steringer: agree with Pam; 75% is above and beyond intent of law Review of written comment process
Sarah Kowalski: minutes will be provided to the RAC, if individuals have feedback and comments after this RAC there will be an opportunity to submit additional comments by a deadline provided ahead of the next RAC. All comments will be distributed to all RAC members. This is not a public comment period, there will be a defined public comment period at a different point after the last RAC, please see timeline in materials for clarification.
Review of statement of fiscal impact
Comments:
Sarah Kowalski: Reviewed language on statement of fiscal impact, asked RAC to review language and make comments and provide feedback.
Public Comment Two individuals signed up to provide public comment
43
Comments:
Toby Absher: will provide written response at later date
Doug Barber (lobbyist with Willamette Dental): goal for legislation is to encourage and facilitate pilot projects; these rules are trying to limit projects and isn’t fair. Need to figure out what is best model for Oregon and the only way to do that is have pilot projects to see what can be learned. The two key components of this rule should both go- 75% and retroactivity. Intention was to be loose and encourage variety in projects. Future rules should not be applied retroactively.
Next Meeting: Monday, October 28, 2019 Portland State Office Building, 800
NE Oregon Street Portland, Oregon, Room 900, 9:00 – 11:00 am
D entists, as doctors of oral health, oversee clinical teams to en-sure the delivery of “safe and effective oral care.”1 Emerging
scientific literature,2–11 however, suggest that dental patientsexperience a significant number of adverse events (AEs) or un-necessary harm while receiving dental care, such as tooth crowningestion or aspiration, wrong tooth extraction, or unexpected se-vere and prolonged pain after molar extractions. Providing safeoral care implies reducing the risk of inflicting unnecessary harmto the dental patient to an acceptable minimum.7 Harm refers to
any “impairment of structure or function of the body and/or anydeleterious effect arising there from.”12 The patient safety para-digm13 starts with the proper identification and assessment ofAEs in a professional culture open to learning from mistakes.14
The Agency for Healthcare Research and Quality developed adetailed patient safety initiative with a goal to “have a positive im-pact on patient safety by providing knowledge and tools to under-stand medical errors and to create solutions that mitigate oreliminate harm to patients suffered as a result of health care.”13
To the best of our knowledge, specific dental-related patient safetymetrics are yet to be developed. To fill this gap, the authors ob-tained grant funding (NIDCR 1R01DE022628-01A1) to developa patient safety initiative for dentistry.
In addition, other healthcare industries such as the pharmaceu-tical and medical device research industries have mandatoryreporting requirements for clinical research. When AEs occur,they systematically document the seriousness of the AE (level ofharm), its impact on enrolled participants, and its association witha study related device, drug, or procedure. This enables the iden-tification of the various contributing factors and allows for thecreation and dissemination of recommendations for systemschanges.15 By contrast, clinical dentistry does not have anysuch mandatory reporting requirements for AEs, and if wedid, there would be no standardized format for reporting theseevents. A dental AE classification system would help better or-ganize and communicate about the types of harm in the dentaloffice. It would provide insights into their prevention, elimination,and/or mitigation of their effects. The impact of AEs is also notequal, and some cause greater harm than others; therefore, a stan-dardized severity rating is needed to understand the extent of dam-age caused byAEs. In the absence of any precursory dental-specificmetrics and tools, we turned to systems developed by the medicalprofession for classifying, assessing severity and reporting AEs.
The National Cancer Institute's (NCI) Common TerminologyCriteria for Adverse Events v4.0 (CTCAE) is a comprehensivecategorization system of AEs in cancer treatment that includes aseverity grading scale for AEs.16 It uses terms taken from the clin-ically validated Medical Dictionary of Regulatory Activities(MedDRA) and is organized across 24 primary system organ clas-ses.16 Another notable classification system was used in theHarvard Medical Practice Study (HMPS), which categorizedhospital AEs according to the type of injury and incorporateda six-point disability scale on which “serious” disability was de-fined as disability persisting for more than 6 months.17 Adverseevents were classified in operative and nonoperative, each con-taining five and ten subcategories, respectively.18 The WorldHealth Organization (WHO) International Classification for Pa-tient Safety (ICPS) is a conceptual framework that consists often high-level classes, each further hierarchically subdividedinto categories and subcategories.12 Forty-eight concepts havebeen identified with agreed upon definitions and preferredterms.12 The degree of harm is defined along five levels from
From the *University of California, San Francisco, School of Dentistry,San Francisco, California; †Simmons School of Social Work, Boston,Massachusetts; ‡Harvard School of DentalMedicine, Boston,Massachusetts;§Oregon Health & Science University, School of Dentistry, Portland, Oregon; ||University of Texas Health Science Center, School of Dentistry at Houston,Houston, Texas; and ¶Center for Biomedical Informatics, Harvard MedicalSchool, Boston, Massachusetts.Correspondence: Elsbeth Kalenderian, DDS, MPH, PhD, Department of
Preventive and Restorative Dental Sciences, School of Dentistry, Universityof California, San Francisco, 707 Parnassus Ave, San Francisco, CA94143-0758 (e‐mail: [email protected]).
The authors disclose no conflict of interest.This research was supported in part by an NIDCR 1R01DE022628-01A1
none to death.12,19 The ICPS is not considered a classificationbut rather a framework with a set of concepts that are linkedby semantic relationships.12,19
Similarly, the Medicare hospital-acquired conditions classifi-cation20 contains ten categories that are mainly surgical and post-surgical management related; however, it does not have a severityrating scheme. The United States National Quality Forum cap-tures the level of harm in serious reportable events.21 As part ofthe Outpatient Adverse Event Trigger Tool developed by the Insti-tute for Healthcare Improvement (IHI),22 a severity classificationmethodology was proposed using the National CoordinatingCouncil for Medication Error Reporting and Prevention (NCCMERP) Index for Categorizing Errors.23 In sum, medical classifi-cation and severity rating systems demonstrate the viability ofmonitoring patient safety, important steps in moving toward a pa-tient safety initiative.4 As expected, our evaluation of these sys-tems quickly revealed that dental AEs do not neatly fit into thecategories developed in the medical realm. Similarly, the level ofseverity of dental AEs did not easily fit within the existing medi-cal severity scales. The focus of this article is to report on themethodology for developing and refining a usable dental AE clas-sification and severity rating system, and the results of a pilotstudy to evaluate its usefulness in classifying AEs found throughchart reviews.
METHODSThe research was reviewed and approved by the human subject
committees of all participating academic institutions.
Development and Refinement of the Dental AEType Classification
The following five medical classifications were analyzed fortheir overlap in categories: (1) NCI's CTCAE16 24 system organclasses; (2) HMPS24 11 categories; and the condensed versionof the HMPS categorization that was introduced byNuckols et al24
with 10 broad categories. (3) The WHO's ICPS12,19 13 categorieswithin the “incident type” class, (4) IHI outpatient trigger tool's22
11 categories, and (5)Medicare's Hospital-Acquired Conditions'20
ten categories. After detailed review of the five classificationschemes, we concluded that they lacked pertinence to dental med-icine, and hence, we handpicked by consensus some of the criteriawith input from our advisory committee.
The initial dental AE type classification (comprising 23 cat-egories) was developed by analyzing dental AEs reported to theFDA MAUDE database2 and documented in the scientific litera-ture.25 This work produced a dental AE list that was expanded af-ter collecting a list of commonly encountered AEs from dentalproviders.26 The initial dental AE type classification and the ag-gregated findings from our comprehensive analysis of medicalAE classification systems were reviewed by the research group'sadvisory committee, comprising experts in medical AEs (see Ac-knowledgments). The findings from each preceding stage of thisprocess led to the creation of aworking system for classifying den-tal AEs. This classification was then pilot tested by independentreviewers across the four sites using a focused chart review pro-cess. This led to the further refinement of the AE classificationand severity systems. For example, during the calibration processfor the chart reviews, we discovered that sinus perforation (a fre-quently reported AE in our previous study26) could be classifiedas either a soft tissue injury, or a hard tissue injury. As a result,we created an additional classification for AEs that did not fit intoa single existing category, other orofacial harm. We also droppedthe use of the word “complication” and replaced it with “harm.”The last 3 of the 12 AE classification categories in Table 1 are
now “other orofacial harm,” “other systemic harm,” and “otherharm.” Finally, all prospective AE cases were verified collectivelyusing a consensus process during conferences calls and a full-dayin-person working meeting.
Development and Refinement of the Dental AESeverity Classification
To our knowledge, there is no standardized measure forassessing the severity of dental AEs. To develop a severity scalefor the AE classifications, we systematically reviewed the severityratings of AEs used in the IHI outpatient trigger tool, NCICTCAE, WHO ICPS, and National Quality Forum.
The IHI's outpatient trigger tool22 has five categories of harm(from least to most severe): temporary harm to the patient and re-quired intervention, temporary harm to the patient and requiredinitial or prolonged hospitalization, permanent patient harm, inter-vention required to sustain life, and patient death. The CTCAE16
assesses the severity of an AE through five gradients of harm(from least to most severe): (1) mild, asymptomatic or mild symp-toms, clinical or diagnostic observations only, and intervention notindicated; (2) moderate; minimal, local, or noninvasive interven-tion indicated, limiting age-appropriate instrumental activities ofdaily living; (3) severe or medically significant but not immedi-ately life-threatening, hospitalization or prolongation of hospitali-zation indicated, disabling, and limiting self-care activities of dailylivings; (4) life-threatening consequences and urgent interventionindicated; and (5) death related to AE. The WHO ICPS used afive-point gradient for assessing the degree of harm: none, mild,moderate, severe, and death.
Using the findings from our review of the medical AE sever-ity ratings and feedback from our advisory committee, we createdan initial AE severity rating scale, which was used to assess the se-verity of AEs in our previous work25 and modified in subsequentwork.3 On the basis of our observations in these studies andthrough an iterative process, we further refined the severity scaleand created a severity tree to simplify its use in the chart reviewprocess (Fig. 1).
Pilot Test (Chart Review Process)Eight independent research team members representing four
United States academic dental institutions (two per site) per-formed focused chart reviews22 using newly constructed or previ-ously developed triggers3 of active electronic health records.
TABLE 1. Dental AE Type Classification
AE Category AE Count
Pain 56
Infection 17
Hard tissue damage 12
Nerve injury 6
Soft tissue damage/inflammation 5
Other orofacial harm 2
Allergy, toxicity, or foreign body response 1
Aspiration or ingestion of foreign body 1
Wrong site, wrong patient, or wrong procedure 0
Bleeding 0
Other systemic harm 1
Other harm 0
Total 101
Kalenderian et al J Patient Saf • Volume 00, Number 00, Month 2017
Please see Table 2 for examples of dental triggers and the AEsthey are intended to detect (Table 3). A “trigger” is an opportunityor clue used to identify AEs in a patient's dental record but do notrepresent AEs themselves. The eight reviewers were tasked withdetermining whether the case fit the definition of an AE. The out-come of interest was the presence of an AE, which was coded aseither Yes or No. If anAEwas present, we next categorized the AEtype and severity. A standardized log sheet was developed to ex-tract the AEs from the charts. The reviewers were trained and cal-ibrated using a uniform AE definition, classification (AE type),and level of harm (AE Severity). Interrater reliability was calcu-lated using the prevalence and bias-adjusted κ to address the κ
paradox. The average percent agreement for AE determinationwas 82.2%. Furthermore, the average, pairwise prevalence, andbias-adjusted κ (PABAK) was 57.5% (κ = 0.575) for determiningAE presence. The average percent agreement for categorizationof the AE type 78.5%, whereas the PABAK was 48.8%. Lastly,the average percent agreement for categorization of AE
severity was 82.2% and the corresponding PABAK was71.7%. According to the standards for interrater reliability, aκ ranging from 0.40 to 0.60 constitutes moderate agreement.27,28
All statistical calculations were performed in R v3.1.1 using the“irr” and “epiR” packages.
RESULTS
Dental AE Type ClassificationA comparison of the five medical AE classifications showed
an overlap of concepts in the surgical/medical procedure, generaldisorders, and infection categories (Appendix 2). However,none of these schemes work well for outpatient dental medi-cine. For example, categories to indicate damage to hard oral tis-sues, e.g., teeth, were difficult to categorize using any of theseexisting schemes.
Suggestions that came from the medical AE experts on theAdvisory committee were critical. Based on their early experi-ences developing medical classification systems, they suggestedtesting the clinical validity of any given AE with the “give-me-a-break” test. That is, to label an event an AE, it must stand up tothe rigor of peer review by professional colleagues. For example,would the failure of a provisional crown constitute an AE? Ini-tially, we thought yes, but although it would be undesirable to havea provisional crown fail, a singular failure did not pass this test. Onthe other hand, if the provisional crown failed time and againwas aspirated or led to an abutment tooth fracture, it would beconsidered an AE. A similar example in medicine would bevomiting after chemotherapy, which the advisory committee
FIGURE 1. Dental AE severity tree. Description of dental AE severity categories: category E1: temporary (reversible or transient) minimal/mildharm to the patient. Category E2: temporary (reversible or transient) moderate to severe harm to the patient. Category F: harm to thepatient that required transfer to emergency room and/or prolonged hospitalization. Category G1: permanent minimal/mild patient harm.Category G2: permanent moderate to severe patient harm. Category H: intervention required to sustain life. Category I: patient death.Severity tree showing the chart review process for assigning severity categories to an AE. The reviewer begins on the left side and followsthe branches of the tree to the right by answering each question.
TABLE 2. Dental AE Severity Classification
AE Severity Count
E2 (temporary moderate to severe harm) 89
G2 (permanent moderate to severe harm) 10
E1 0
G1 1
Total 101
J Patient Saf • Volume 00, Number 00, Month 2017 Classifying Adverse Events in the Dental Office
explained was not considered an AE in itself, unless ongoing vio-lent vomiting resulted in an inability to absorb nutrients and re-quiring parental nutrition/rehydration.
Putting together our findings from the analysis of these fivemedical AE classifications, the dental AEs found through theFDAMAUDE database,2 our literature review,25 our empirical in-terviewswith providers,26 our consultationwith the advisory com-mittee on this project, and our focused chart reviews, we maderevisions to the initial dental AE classification system26 and ar-rived at 12 final categories for the Dental AE Type ClassificationSystem. Please see Table 1.
Dental AE Severity ClassificationIn reviewing the four medical severity ratings, we found that
although they effectively reflected increasing degrees of severitybased on the temporal impact of harm and what was needed tomit-igate the effects of the AE, it was not fully applicable to outpatientdentistry. Adverse events in dentistry seem to be less catastrophic,and as such, we felt it necessary to be able to differentiate not onlybetween temporary and permanent harm but indicate if the harmwas mild or severe.
Specifically, we noted that the CTCAE severity grades,ICPS, and the IHI scales had some similarities (death, interventionrequired). They also had relevance for oral health. Specifically,the IHI scale had utility for dentistry. It assessed harm on the ba-sis of the short- and long-term impact of the AE upon the patient.The more severe the immediate impact or more extensive thelong-term mitigation required, the higher the severity rating.We used this approach in developing our own severity scale fordental AEs.
Items from the IHI trigger tool, ICPS, and the CTCAE wereintegrated into more granular elements specific to oral health.With the support of the advisory board, we developed an initialAE severity scale for oral health comprising 15 items. This scalewas pilot tested in our previous work analyzing the scientific liter-ature.25 Based on feedback from the reviewers and through an it-erative process, it was further condensed, simplified, and adaptedinto a severity tree (Fig. 1). The first four items on the scale (A–D)were dropped, the “magnitude of the intervention”was also droppedfrom each step, and the “moderate” and “severe” categories werecombined. The final step was the application of the severity scale
to AEs identified through electronic health record chart reviewsby independent reviewers across several sites.
Overall Evaluation of the Dental AE Type andSeverity Classifications
The following shows an example of a case that a reviewerwould be asked to classify:
“While a gold onlay for #30 was being tried in prior to ce-mentation, the onlay inadvertently became dislodged and lost inthe oropharyngeal space. KUB revealed a radiopaque foreign ob-ject in the area of the duodenum, measuring approximately 1 cm.Patient informed that her airways were clear and that she will passthe foreign body.”
Reviewers would classify the above as AE type: “aspiration/ingestion of foreign body” with severity of temporary (reversibleor transient) moderate to severe harm to the patient (E2). Re-viewers continue to wrestle with adjudicating severity. In thiscase the fact that the patient underwent a KUB was consideredan invasive sequela and moved the severity to moderate. Thiscase leans towards the moderate spectrum, however, as moder-ate and severe are combined into the E2 severity scale, it wasclassified as E2. A more severe example would be if the crownhad been aspirated.
There were 3283 (not including random charts) triggeredcharts. In total, there were 3283 charts triggered for all triggersacross 4 participating dental institutions. For each trigger a samplesize of up to 50 charts were reviewed per site, which led to a sam-ple size of 958 charts. Among the reviewed charts, 118 prospec-tive AEs were found and 101 (85.6%) were verified as AEsduring the consensus process. Pain and infection were the mostcommon AE types representing 75% of the cases reviewed (55%and 17%, respectively). In the remaining reviews, hard tissue dam-age was assessed in 12%, soft tissue damage/inflammation in 6%,nerve injury in 5%, and other orofacial harm in 2% of cases.Examples of AEs found during the chart reviews include thefollowing: dry socket, failure of implant to osseo-integrate2 months after placement with loss of bone, and requiring re-moval; pulp exposure during caries removal due to suddenmovement of pediatric child; pain; and excessive swelling. Re-sults of the classification after consensus was reached are doc-umented in Table 1. Overwhelmingly, the most recorded AE
TABLE 3. Dental Triggers Showing Reviewed Charts (3283Charts Were TriggeredWith Specific Triggers and 91,936With a RandomSample of Charts)
Triggers No. Triggered Charts No. Reviewed Charts
T1: Extraction after RCT/crown/filling 110 99
T2: Untreated periodontitis 224 100
T3: Failed implant 34 34
T4 : Postsurgical extraction complications or Post Perio TX complications 377 100
T5: Repeated fillings 391 129
T6: Multiple visits 60 58
T7: Random charts 91936 99
T8 : Nerve injury 36 36
T9: Infections 430 100
T10: Soft tissue injury/inflammation 1449 100
T11: Allergy/toxicity/foreign body response 36 35
T12: Aspiration/ingestion of foreign body 136 68
Total 3283 (+91936) 958
RCT, root canal treatment.
Kalenderian et al J Patient Saf • Volume 00, Number 00, Month 2017
severity was “temporary, moderate to severe harm to the patient”(E2) representing 88% of the cases (Table 4).
DISCUSSIONThe medical profession has made considerable strides under-
standing patient safety. It is now time for dentistry to embrace pa-tient safety and move toward a better safety initiative.4,29 Tocreate, monitor, and maintain what the Agency for Healthcare Re-search and Quality describes as a patient safety initiative,13 theidentification and assessment of AEs are important first steps.The AE classifications and severity ratings provide unique oppor-tunities to the dental profession to explore how to provide safe andeffective high-quality oral care to patients. The nature of AEs thathave been reported in the medical literature is different from thosethat occur in dentistry. Significant AEs in the dental office are rareand seldom life-threatening. In addition, with 32 teeth as a startingpoint and our ability to function wellwith significantly fewer teethas well as our ability to replace lost teeth, the attitude toward acci-dently injuring a tooth has been quite different from doing sowithany other body part. Our results suggest the feasibility of the useof a classification system in helping to organize the different typesof AEs that patients may encounter through dental treatment. It isimportant to realize the difference between harm and contributingfactors that may lead to harm, e.g., aspiration of a gold onlay is theactual harm, whereas not using a rubber dam or unexpectedmove-ment of the patient would be contributing factors.
There were challenges in classifying some of the AE cases weencountered in our chart review. In our study, the reviewers were
asked to pick the single best category to describe the AE. How-ever, we discovered that some AEs could be classified into mul-tiple categories, e.g., a patient presenting with swelling andsignificant pain 2 days after periodontal flap surgery could beclassified under “pain” as well as “infection.”Although restrictingthe classification to only one category is useful for reporting pur-poses, this approach may not fully capture the nature of the harm,which is a limitation of our approach. In some cases, our chart re-viewers reported that there was insufficient information to classifyan AE, e.g., radiographs could be helpful to determine if aperiapical abscess is new or predated restorative treatment. Thisspeaks to the importance of having adequate clinical documentationthat can be used to assess the quality and safety of dental care.30
The inclusion of pain as an AE may be considered problematicas some will suggest that it is difficult to objectively confirm theharm. Indeed, one may argue that dental pain is often a conse-quence of dental treatment. We discussed this at great length withthe advisory board and developed guidelines for when expectedpostprocedure pain would rise to the level of an AE. We under-stood that pain will vary greatly depending on the procedure anda number of patient factors. However, we agree with Chorneyand et al31,32 that “inadequately managed pain meets the defini-tion of an AE and… that treating pain as an AEmay improve careby raising… awareness of this issue.”32 We also understand thatthe practical application of our proposed classification systemwill be challenging as the dental profession is not accustomedto evaluating and tracking the pain experience, as is commonplacein the medical profession (e.g., postoperative pain scales, etc.).Therefore, it is critically important that as an AE classification
TABLE 4. Dental Triggers Examples
Trigger Name Trigger Description AEs Detected
Allergy or toxicity orforeign body response
Patients who had “foreign body” textin their notes and had received at leastone treatment in the given calendar year
Allergic reaction to orthodontic brackets,or medication
Aspiration or ingestionof foreign body
Patients who had terms like “aspiration,”“aspirated” in their notes and had receivedat least one treatment in the given calendar year.
Ingestion or aspiration of crown or screwduring placement of restoration
Failed implant Patients who had a failed implant diagnosisor implant removal procedure code on anytooth in the given calendar year.
Peri-implantitis, lack of implantintegration
FIGURE 2. Patient safety is a core component of quality of care. (Institute of Medicine (2000). To Err Is Human36). A hypothetical illustrationof safety as a component of quality dental care delivery using tooth crowns. The smallest circle represents the attempt to keep the patientfree from accidental injury by ensuring the patient does not aspirate the crown. This fits into the bigger circle of quality by ensuring the crownis functional. The last piece of quality is to ensure that it meets the patient's preference and aesthetic expectations.
J Patient Saf • Volume 00, Number 00, Month 2017 Classifying Adverse Events in the Dental Office
scheme is developed in dentistry, and the triggers to classify anoutcome as adverse must be objective and unambiguous. Be-cause the importance of the pain experience cannot be over-estimated when patient values, expectations, and preferencesare considered, we have embarked on further exploring our painAEs to see how we can better understand underlying systemimprovement opportunities.
We have not yet explored diagnostic failures,33,34 including de-layed diagnosis, missed diagnosis, and misdiagnosis as an AE, asour focus has been on physical harm due to commission. In futurework, we will expand our work to include diagnostic failures.
Our severity scale was adapted from one developed by NCC-MERP to classify medication-related AEs.23 The severity of harmin dentistry is qualitatively different from that in medicine. Al-thoughmedicine is focused on cases of severe harm (such as deathor requiring hospitalization), the most harm that occurs in den-tistry is less life altering. Hence, we not only elected to captureharm that is either permanent (extraction of the wrong tooth) ortemporary (sinus perforation) but also further divided the harminto slight or moderate/severe in an effort to better distill the mostsevere cases. We did not explore cases indicated as slight or min-imal harm as we believe that in this first effort, focusing on moresevere harm will help us ultimately undercover underlying sys-tems that can be improved to prevent these more extreme formsof harm from happening again.
The patient safety revolution can be traced to the seminal Insti-tute of Medicine seminal report, “To Err Is Human.” It states thatquality consist of the following three domains: (1) safety, definedas “freedom from accidental injury”; (2) practice consistent withcurrent medical knowledge and best practice; and (3) responsive-ness to customer-specific values, expectations and preferences.35
We have visually presented these concepts contextualized for thedental profession in Figure 2. All of these elements must be metto achieve quality. Assessing adherence to best practices such aspercentages of patients having annual dental visits is an importantmarker, but not a substitute for assessing safety. There must bemarkers to assess patient safety so that trends can be observed, re-ported, and used to improve quality.
Reporting of AEs is a crucial step for any organization orprofession to learn from its mistakes and move toward the estab-lishment of a learning organization37 or profession. Reportingof AEs, however, does not improve safety in and of itself. AnAE must be much more than a report. It should lead to exploringunderlying systems failures, ultimately leading to change.15
Individuals as well as organizations will gain more fromreporting AEs when their information is aggregated and com-pared with others so that learning can occur across settings to pre-vent or minimize the probability of recurrences of the same orsimilar AEs.15
Classifying AEs, categorizing their severity, and eventuallystandardizing how AEs are captured in databases for query arekey factors to the development of a learning profession. Medicinehas accomplished many of these tasks. Although dentistry hasonly begun embracing a patient safety paradigm, it does not haveto take the long road that our medical colleagues have traveled.We can learn from their triumphs and strive toward the creationof a learning profession by not only agreeing that patient safetyis the first element of quality but also adopt a standardized classi-fication of AEs and level of harm as a crucial ingredient in the de-velopment of this endeavor.
CONCLUSIONSPatient safety is a critical component of quality, and classify-
ing AEs and their severity is an important step toward the ability
to analyze patient safety data in a meaningful way. The use ofdental AE type and severity classifications facilitate the categori-zation of and communication about dental AEs during routinechart reviews.
ACKNOWLEDGMENTThe authors thank Dr. Lucian Leape for the unwavering
support and advice and Advisory Committee members includingDrs. Eric Thomas, Joan Ash, John Valenza, Debora Simmons,Ana Karina Mascerenhas, Roger Resar, Ms. Linda Kenney andMr. Michael Cohen for the invaluable feedback.
REFERENCES
1. AmericanDental Association Dentists: Doctors of Oral Health Chicago, Ill:
American Dental Association 2015. Available at: http://www.ada.org/en/
about-the-ada/dentists-doctors-of-oral-health. Accessed June 30, 2015.
2. Hebballi NB, Ramoni R, Kalenderian E, et al. The dangers of dental
devices as reported in the Food and Drug AdministrationManufacturer and
User Facility Device Experience Database. J Am Dent Assoc. 2015;146:
102–110.
3. Kalenderian E, Walji MF, Tavares A, et al. An adverse event trigger tool in
dentistry: a new methodology for measuring harm in the dental office.
J Am Dent Assoc. 2013;144:808–814.
4. Ramoni R, Walji MF, Tavares A, et al. Open wide: looking into the safety
culture of dental school clinics. J Dent Educ. 2014;78:745–756.
5. Ramoni RB, Walji MF, White J, et al. From good to better: toward a patient
safety initiative in dentistry. J Am Dent Assoc. 2012;143:956–960.
6. Thusu S, Panesar S, Bedi R. Patient safety in dentistry - state of play as
revealed by a national database of errors. Br Dent J. 2012;213:E3.
7. Bailey E, Tickle M, Campbell S. Patient safety in primary care dentistry:
where are we now?. Br Dent J. 2014;217:339–344.
8. Hiivala N, Mussalo-Rauhamaa H, Murtomaa H. Patient safety incidents
reported by Finnish dentists; results from an internet-based survey.
Acta Odontol Scand. 2013;71:1370–1377.
9. Hiivala N, Mussalo-Rauhamaa H, Tefke HL, et al. An analysis of dental
patient safety incidents in a patient complaint and healthcare supervisory
database in Finland. Acta Odontol Scand. 2015:1–9.
10. Hiivala N, Mussalo-Rauhamaa H, Tefke HL, et al. An analysis of dental
patient safety incidents in a patient complaint and healthcare supervisory
database in Finland. Acta Odontol Scand. 2016;74:81–89.
11. Bailey E, Tickle M, Campbell S, et al. Systematic review of patient safety
interventions in dentistry. BMC Oral Health. 2015;15:152.
12. WorldHealth Organization. The conceptual framework for the international
classification for patient safety. World Health Organization. 2009;2009:
1–149.
13. Agency for Healthcare Research and Quality. Chapter 3. AHRQ's Patient
Safety Initiative: Breadth and Depth for Sustainable Improvements. In:
AHRQ's Patient Safety Initiative: Building Foundations, Reducing Risk.
Rockville, MD: Agency for Healthcare Research and Quality; 2003.
14. Yamalik N, Perea Pérez B. Patient safety and dentistry: what do we need to
know? Fundamentals of patient safety, the safety culture and
implementation of patient safety measures in dental practice. Int Dent J.
2012;62:189–196.
15. World Alliance for Patient Safety WHO Draft Guidelines for Adverse
Event Reporting and Learning Systems. From Information to Action.
Geneva, Switzerland: WHO Press 2005. Available at: http://www.who.int/
Prolonged (e.g., not rapidlyresponsive to symptomaticmedication and/or briefinterruption of infusion);recurrence of symptomsfollowing initial improvement;hospitalization indicated forclinical sequelae (e.g.,renal impairment,pulmonary infiltrates)
Asymptomatic or mildsymptoms; clinical ordiagnostic observationsonly; interventionnot indicated
Moderate; minimal,local or noninvasiveintervention indicated;limiting age-appropriateinstrumental ADL
Severe or medicallysignificant but notimmediately life- threatening;hospitalization orprolongation of existinghospitalization indicated;disabling; limitingself care ADL
Asymptomatic or mildsymptoms; clinicalor diagnostic observationsonly; interventionnot indicated
Moderate; minimal,local or noninvasiveintervention indicated;limiting age-appropriateinstrumental ADL
Severe or medically significantbut not immediatelylife-threatening; hospitalizationor prolongation of existinghospitalization indicated;disabling; limitingself care ADL
Hypertension Prehypertension(systolic BP120–139 mm Hgor diastolic BP80–89 mm Hg)
Stage 1 hypertension(systolic BP140–159 mm Hg ordiastolic BP90–99 mm Hg);medical interventionindicated; recurrentor persistent (≥24 hours);symptomatic increaseby >20 mm Hg (diastolic)or to >140/90 mm Hgif previously WNL;monotherapy indicatedPediatric: recurrent orpersistent (≥24 hours)BP >ULN; monotherapyindicated
Stage 2 hypertension(systolic BP ≥160 mm Hgor diastolic BP ≥100 mm Hg);medical intervention indicated;more than one drug or moreintensive therapy thanpreviously used indicatedpediatric: same as adult
Life-threatening consequences(e.g., malignant hypertension,transient or permanentneurologic deficit,hypertensive crisis);urgent interventionindicated pediatric:same as adult
H ealth care is a high-risk industry,1 and monitor-ing the quality and safety of care provided in one’s
dental practice is central to pro-viding high-quality dental care. Despite this, there have been no tools developed, to our knowledge, to assist dental care teams in ef-ficiently and effectively conducting this fundamental task. Perhaps as a consequence, there is a dearth of information about the occurrence of adverse events (AEs) in the dental office. Significant AEs primarily are reported as case studies. These reports indicate the breadth of harm that can occur in the dental clinical setting, including deaths2-5; severe (life-threatening) events such as life-threatening airway obstruction6,7; severe (non–life-threatening) events, such as permanent nerve damage8,9; and less severe events, such as an allergy to nickel in an orthodontic patient10 or extraction of the wrong tooth,11 which are rectifiable. Thus, the existing literature supports the need for tools to help dental prac-tices perform regular safety and quality assessments.
The Institute for Healthcare Im-provement (IHI), Cambridge, Mass., developed trigger tools to address
Dr. Kalenderian is chair, Oral Health Policy and Epidemiology, Chief of Quality, Office of Clinical Affairs, Harvard School of Dental Medicine, 188 Longwood Ave., Boston, Mass. 02115, e-mail [email protected]. Address reprint requests to Dr. Kalenderian.Dr. Walji is an associate professor, Diagnostic and Biomedical Sciences, and director of informatics, Office of Technology Services and Informatics, School of Dentistry, University of Texas Health Science Center at Houston.Dr. Tavares is a research assistant, Harvard School of Dental Medicine, Boston. Dr. Ramoni is an instructor in pediatrics and executive director, SMART Project, Center for Biomedical Informatics, Harvard Medical School, Boston.
An adverse event trigger tool in dentistryA new methodology for measuring harm in the dental office
Elsbeth Kalenderian, DDS, MPH; Muhammad F. Walji, PhD; Anamaria Tavares, DDS; Rachel B. Ramoni, DMD, ScD
Background. There is a dearth of knowledge about the type and frequency of adverse events (AEs) in dentistry. Current approaches to ob-taining information rely on reviews of randomly selected records, which may not be the most efficient or effective methodology.Methods. Inspired by the Institute for Healthcare Improve-ment’s (IHI) global and outpatient trigger tools, which identi-fies records with characteristics (“triggers”) that are associated with AEs, the authors created the dental clinic trigger tool. The triggers included procedures for incision and drainage, failed im-plants and selected treatment patterns. The authors ran the trig-ger tool against six months of electronic health records data and compared its performance with that of a review of 50 randomly selected patient records. Results. In total, 315 records were triggered, 158 (50 percent) of which were positive for one or more AEs; 17 (34 percent) of the 50 randomly selected records were positive for at least one AE. The authors assigned each AE an IHI severity ranking. Most AEs caused temporary harm, but nine were considered to have caused permanent harm according to a modified IHI severity ranking.Conclusions. The study results demonstrate the promise of a directed records review approach, as the dental clinic trigger tool was more effective in identifying AEs than was a review of randomly selected records.Practical Implications. All dental practices should proactive-ly monitor the safety of the care they provide. Use of the trigger tool will help make this process more efficient and effective.Key Words. Patient safety; trigger tool; adverse event; den-tistry; electronic health record.JADA 2013;144(7):808-814.
documented patient care completely on paper, except for billing information and scheduling processes. Staff members transferred 24 months of billing data for active patients (those who had been seen within the last 24 months) into the EHR system; they did not transfer or scan any other data into the system. The EZCodes version 2012 dental diagnostic terminology20 (president and fellows of Harvard College, Boston; Board of Regents of The University of Texas System, Houston; Academic Centre for Dentistry Amster-dam, Netherlands; and Regents of the Universi-ty of California, San Francisco) is the diagnostic terminology in place at the study site.
Dental clinic trigger tool. The dental clinic trigger tool includes the following three triggers framed to gain insight into AEs that are related to underlying systemic issues: development of infections (“incision and drainage”), failure of complex procedures (“implant failure”) and handoff (“multiple visits,” defined in the next paragraph). We chose these three triggers be-cause they related to triggers used by the IHI’s Global Trigger Tool and could be queried in the EHR through the use of standardized Current Dental Terminology (CDT) codes.21
We defined the incision and drainage trigger as completion of the procedure described in CDT code D7510 (incision and drainage of abscess–intraoral soft tissue) or in D7520 (incision and drainage of abscess–extraoral soft tissue).21 We defined the implant failure trigger as comple-tion of the procedure described in CDT code D6100 (implant removal, by report) or in EZCodes 563101 or 977294 (failing implant or peri-implantitis). We defined the multiple-visits trigger as having fulfilled any of the following during the six-month review period: having had more than six completed visits; being seen by more than one general dentist or more than one prosthodontist or by a general dentist and a prosthodontist; being seen by a general dentist or prosthodontist and an endodontist; or having had three or more visits to an endodontist. Our rationale behind the design of the last trigger was that most treatment plans are completed within six months in the FP; being treated by more than one of the same type of provider (gen-eral dentist or prosthodontist for general dental care) might indicate that the patient was seeing another provider owing to an emergency; and
the fact that expensive and less effective meth-ods (such as audits of patient records, voluntary reporting by providers and direct observations) had been used in medicine to identify AEs.12-15 A trigger is an easily identifiable focused item in a patient record that can help lead to the identification of an AE. Triggers serve to alert reviewers who are looking for AEs in a sample of patients’ records to focus further investigation on a subset of triggered records to determine whether an AE actually occurred.16 For instance, administration of naloxone, a strong narcotic antagonist, often indicates overdosage, except in the case of drug abuse or a self-inflicted overdose.
The IHI initially developed global16 and specialty-area17 (for example, intensive care unit, mental health) tools for use in inpatient settings. Classen and colleagues18 reported that the Global Trigger Tool16 detected 10 times more AEs than did other approaches. In parallel, IHI developed an Outpatient Adverse Event Trigger Tool, with 11 triggers, which was tested at Kaiser Permanente and Baylor Health System in 2005 and 2006.19
Although useful in the medical outpatient set-ting, the Outpatient Adverse Event Trigger Tool would not be directly applicable to the dental clinical setting. For example, the trigger “abnor-mal laboratory value” is not broadly relevant in the dental care setting (Table 119). Thus, we took our inspiration from the Outpatient Adverse Event Trigger tool to create a dental clinic trigger tool and to compare its performance with that of a review of randomly selected patient records.
METHODSAt the study site, patients can obtain dental care from predoctoral dental students and advanced graduate dental residents in the teaching prac-tices (TPs) or from academic faculty members and hygienists in the faculty practice (FP), who provide care in a private group practice setting. Patients receive comprehensive care in general dentistry, periodontics, prosthodontics, implant dentistry, endodontics, orthodontics, oral surgery and dental hygiene. Only predoctoral dental stu-dents in years 3 and 4 of training treat patients in the TP. Dental students are overseen in the TP by full-time academic and adjunct (volun-teer) faculty members. The FP is a completely separate practice in which 20 full-time faculty members provide care.
Electronic health record. In 2009, the TPs and FP implemented use of an electronic health record (EHR) (axiUm, Exan, Coquitlam, British Columbia, Canada). Before then, the practices
ABBREVIATION KEY. AE: Adverse event. CDT: Cur-rent Dental Terminology. EHR: Electronic health record. FP: Faculty practice. ID: Identification. IHI: Institute for Healthcare Improvement. TP: Teaching practice.
eight patients from a list of patients seen during that month, sorted according to patient identi-fication (ID) number. The ID number had been assigned to the patient when he or she registered within the EHR at the study site. Because the study site’s clinics transitioned to an EHR in 2009, both recent and long-standing patients had relatively new ID numbers. For the month of De-cember, we added one additional FP patient and one additional TP patient to arrive at 50 unique records. We selected 50 records because this number afforded us sufficient statistical power to detect a difference of at least 10 percentage points by using the c2 test. Each record repre-sented the complete EHR of a single patient.
As recommended by the IHI methodology,16 two dentists (E.K., A.T.), who were experienced in reviewing patient records for the presence of AEs, independently reviewed each record. Also, in accordance with the IHI trigger tool specifi-cations, we defined an AE as “harm caused by medical treatment, regardless [of] whether it is associated with error or considered preventable.
… It is from the point of view of a patient that harm can sometimes be easily ascertained: ‘If I were the patient, would I be happy if this happened to me?’ ”19(p3) The two dentists reviewed each record systematically by inspecting sec-tions relating to the following: ddiagnoses indicating an AE (for example, “failed implant,” “peri-implantitis”); dtreatments or procedures in the EHR designated by CDT codes21 that indicated a possible AE (that is, CDT D6100 [implant removal], CDT D7510 [incision and drain-age of abscess]);dnarrative/progress notes indi-cating an AE. If an AE was detected, each re-
viewer assigned a severity rating to the record; we used the same severity classification as that used in the IHI Outpatient Adverse Event Trig-ger Tool,19 as shown in Table 2. The IHI severity index was adapted from the National Coordinat-ing Council for Medication Error Reporting and Prevention Index for Categorizing Errors.22 After reviewing the records independently, the two dentists compared their results and resolved any discrepancies.
Finally, we determined the positive predictive value (PPV), which is the proportion of triggered records that had one or more AEs associated with each trigger, as well as the performance
multiple visits for general dental care that are followed by an endodontic visit may indicate pulpal exposure.
We implemented these triggers as a set of queries, which were run against EHR data from the study site’s TPs and FP from July 1, 2011, through Dec. 31, 2011. One of us (E.K.) then reviewed the retrieved records to remove dupli-cates and records that had been identified falsely as being positive for one or more triggers. For comparison, we chose 50 records randomly from the complete set of patients seen by clinicians in the TPs and FP during the same six-month review period. Every month, we chose the first
TABLE 1
Institute for Healthcare Improvement Outpatient Adverse Event Trigger Tool.*TRIggER NuMBER
DESCRIPTION
1 New diagnosis of cancer
2 Nursing home placement
3 Admission and discharge from the hospital
4 Two or more consultants in a year of review
5 Surgical procedure
6 Emergency department visit
7 More than five medications
8 Physician change
9 Complaint letter
10 More than three nursing calls in one week
11 Abnormal laboratory value
* Adapted with permission of the Institute for Healthcare Improvement from the Institute for Healthcare Improvement and Resar.19
TABLE 2
Institute for Healthcare Improvement classification of adverse event severity.*CATEgORy DESCRIPTION ExAMPLE
E Temporary harm to the patient and required intervention
Abscess after endodontic treatment requiring incision and drainage
F Temporary harm to the patient and required initial or prolonged hospitalization
Space infection after third molar surgery with potential for sepsis and airway compromise
g Permanent patient harm Failed implant that was not replaced
H Intervention required to sustain life
None identified
I Patient’s death None identified
* Adapted with permission of the Institute for Healthcare Improvement from the Institute for Healthcare Improvement and Resar.19
the implant failure trigger, 10 indicated one or more AEs, for a PPV of 71 percent. These AEs included diagnoses of failed implants and peri-implantitis, requiring removal or replace-ment of the implant. Finally, we retrieved 287 records from the multiple-visits trigger; 140 of these indicated one or more AEs, for a PPV of 49 percent. These AEs were wide ranging, includ-ing decemented crowns, decemented temporary restorations, alveolar osteitis and a significant needle tract infection after a patient received a mandibular local anesthetic injection. Consid-ered together, being positive for any of the three triggers was associated with a PPV of 50 per-cent. In comparison, the review of 50 randomly selected patient records revealed 27 AEs, for a PPV of 34 percent; this represents a statistically significant difference from the PPV of the trig-ger tool group (c2 test, P = .03). Table 3 shows these results.
Two of us (E.K., R.B.R.) organized the AEs into 10 categories, as shown in Table 4. The most common types of AEs identified according to both methods (that is, triggered selection and random selection) were failed permanent res-torations within five years of placement, failed temporary restorations and inflammation/ infection after dental treatment.
Regarding the severity of the AEs identi-
of the triggers as a whole. We also calculated the proportion of randomly selected records in which the reviewers identified an AE.
RESuLTS A total of 8,931 patients were seen at the study site from July 1 through Dec. 31, 2011. The automated computer queries that were run against the EHR data initially identified 500 triggered records, some of which were dupli-cates because they were captured more than once by the queries. After removing duplicates and records that were incorrectly identified as being positive for the multiple-visits trigger, we had 315 unique and verified trigger-positive records. As mentioned earlier in the Methods section, we also reviewed 50 randomly selected records to compare the AEs identified with those in records retrieved by using the dental clinic trigger tool.
Tables 321 and 4 show the results of the re-cords review. The two reviewers retrieved 14 records through the incision and drainage trig-ger, seven of which indicated one or more AEs, for a PPV of 50 percent. Among the AEs was infection, including abscess formation that re-quired an incision and drainage procedure after endo dontic, periodontic or surgical extraction treatment. Of the 14 records retrieved through
TABLE 3
Dental clinic trigger tool performance.IDENTIFICATION MODE AE* DETECTED NO. OF
RECORDS TRIggERED
NO. OF RECORDS
INDICATINg AE
NO. OF AEs
POSITIvE PREDICTIvE
vALuE (95% CI†)
Incision and Drainage Trigger (CDT‡ D7510, CDT D7520)
* AE: Adverse event. † CI: Confidence interval.‡ CDT: Current Dental Terminology. Source: American Dental Association.21
§ EZCode 563101 (president and fellows of Harvard College, Boston; Board of Regents of The University of Texas System, Houston; Academic Centre for Dentistry Amsterdam, Netherlands; and Regents of the University of California, San Francisco).
¶ Defined as having had more than six dental visits; being seen by more than one general dentist or by more than one prosthodontist, or by a general dentist and a prosthodontist; being seen by a general dentist or a prosthodontist and an endodontist; or having had three or more visits to an endodontist within the six-month period.
to action: as a prac-tice and as a profes-sion, we must strive to do better by the patients who entrust us with their health and safety. The dental profession can and should learn from its mistakes, celebrate its successes and share best practices, as outlined in a recent editorial by Ramoni and colleagues.23 The dental school settings in which we conduct-ed this research are governed by the Com-mission on Dental Accreditation, which states that “the den-tal school must con-duct a formal system of continuous quality improvement for the patient care program that demonstrates evidence of” the fol-lowing: “standards of care that are patient-centered, focused on comprehensive care”; an “ongoing review of a representative
sample of patients and patient records”; “mecha-nisms to determine the cause(s) of treatment deficiencies”; and patient review policies.24
Indeed, all dental care teams should initiate regular assessments of AEs that occur within their practices, including conducting records reviews. Patient safety activities must occur in conjunction with other essential functions of a busy dental practice, so it is our goal to create tools that make these monitoring activities as efficient and effective as possible. The IHI’s trig-ger tools25 pointed the way to a promising ap-proach to accomplish just that.
Inspired by the IHI tools, we created the den-tal clinic trigger tool, a trigger-based approach to AE monitoring in dental practices. Our trig-ger tool corresponds with the following IHI triggers:dThe multiple-visits trigger was based on the IHI physician change trigger (number 8).19 dThe implant failure trigger was based on the IHI surgical procedure trigger (number 5).19
fied via the dental trigger tool, 191 were rated as “E,” because the harm was temporary (for example, extreme pain or abscess formation) but required intervention (Table 2). However, we classified one AE as “F,” which is temporary harm that required hospitalization (a space in-fection after third molar surgery with the poten-tial for sepsis and airway compromise) and nine AEs as “G,” which is permanent harm (for ex-ample, a failed implant that was not replaced). For the randomly selected patient records, we classified all 27 AEs as “E,” because the harm was temporary but required intervention. None of these AEs required hospitalization or were permanent.
DISCuSSIONIn our study population, more than one-third of the randomly selected patients had experienced an AE. Several of the AEs identified via the trig-ger tool necessitated hospitalization or resulted in permanent harm. This baseline is a clear call
TABLE 4
Type and frequency of AEs* identified according to triggered and random reviews of patient records.AE CATEgORy ExAMPLE TRIggER OR
RANDOM SELECTION
NuMBER (%) OF AEs
(n = 228)
Failed Permanent Restoration Within Five years of Placement
Fractured removable partial denture
Multiple visits, random
79 (34.6)
Failed Temporary Restoration
Lost temporary crown
Multiple visits, random
56 (24.6)
Inflammation/Infection After Dental Treatment
Needle-tract infection
Incision and drainage, failed implant, multiple visits, random
54 (23.7)
Iatrogenic Injury Soft-tissue burn due to acid leak during acid etching
Incision and drainage, failed implant, multiple visits, random
detected. As our colleagues at the IHI stated, “The triggers listed in the outpatient trigger tool have been tested but certainly are not the only possible triggers; they represent a good starting point.”19 In the near future, we plan to significantly expand our research in this area: our deliverables will include the creation of an iteratively tested and improved patient safety toolkit for broad use in clinical practice to iden-tify AEs, the estimation of sensitivity and the development of a classification system to catalog AEs consistently.
CONCLuSIONAEs in the dental practice vary from tempo-rary to resulting in permanent harm and can be detected through targeted records reviews. The use of triggers, or clues, to identify AEs from records reviews is a promising method for measuring the overall level of harm to patients from care. Such review of patient records may help dentists understand underlying systemic issues (such as the need for additional training) and becomes an important component of ongo-ing quality improvement efforts. n
Disclosure. None of the authors reported any disclosures.
The authors invite readers to commit to proactively monitoring quality by testing in their own dental practices the dental clinic trig-ger tool, which can be obtained through the corresponding author; by sharing their experiences with the authors; and by suggesting additional triggers.
The authors thank Prof. Lucian Leape for his advice and encour-agement as they developed the dental trigger tool and Scott Jason for his assistance in developing the scripts to run the trigger tools against the electronic health record.
1. Leape L. Lucian Leape and healthcare errors. Interview by Pa-mela K. Scarrow and Susan V. White. J Healthc Qual 2002;24(3): 17-20.
2. Quigley R. Girl, 17, dies during wisdom teeth surgery. Associated Newspapers. Dec. 15, 2011.
3. Davies JM, Campbell LA. Fatal air embolism during dental implant surgery: a report of three cases. Can J Anaesth 1990;37(1): 112-121.
4. Gowans WJ. Fatal methaemoglobinaemia in a dental nurse: a case of sodium nitrite poisoning. Br J Gen Pract 1990;40(340): 470-471.
5. Deegan AE. Anesthesia morbidity and mortality, 1988-1999: claims statistics from AAOMS National Insurance Company. Anes-thesia Prog 2001;48(3):89-92.
6. Bowden JR, Ethunandan M, Brennan PA. Life-threatening airway obstruction secondary to hypochlorite extrusion during root canal treatment (published online ahead of print Oct. 5, 2005). Oral Surg Oral Med Oral Pathol Oral Radiol Endod 2006;101(3):402-404. doi.org/10.1016/j.tripleo.2005.06.021.
7. Weber SM, Chesnutt MS, Barton R, Cohen JI. Extraction of dental crowns from the airway: a multidisciplinary approach. Laryn-goscope 2005;115(4):687-689.
8. Pogrel MA, Thamby S. Permanent nerve involvement resulting from inferior alveolar nerve blocks. JADA 2000;131(7):901-907.
9. Mohammadi Z. Endodontics-related paresthesia of the mental and inferior alveolar nerves: an updated review. J Can Dent Assoc 2010;76:a117.
10. Pazzini CA, Pereira LJ, Marques LS, Generoso R, de Oliveira G Jr. Allergy to nickel in orthodontic patients: clinical and histopatho-logic evaluation. Gen Dent 2010;58(1):58-61.
dThe incision and drainage trigger was based on the IHI emergency department visit trigger (number 6).19
Our study results show that the trigger tool approach is capable of identifying AEs more effi-ciently: 50 percent of records that were positive for any of the three dental triggers contained an AE, whereas 34 percent of randomly selected patient records indicated an AE. Our results in-dicate that use of triggers also may enable clini-cians to identify more severe AEs, though given the relatively low rate of more severe AEs, a sig-nificantly larger sample size would be required to generalize the results of our study.
Although we developed the three triggers as an electronic script and ran it in the context of an EHR, they can be implemented readily in a paper environment. The IHI described exten-sively how to implement a records review in a paper environment.19 In brief, a member of the dental team quickly reviews the records of recently seen patients for the presence of a trig-ger; the records that are positive for a trigger then are subjected to a more in-depth review by at least two people to determine whether an AE actually has occurred.
In the context of the trigger tool, an AE in-volves harm to the patient, regardless of wheth-er the AE is associated with error.19 As defined by the IHI, harm is “unintended physical injury resulting from or contributed to by medical care that requires additional monitoring, treatment or hospitalization, or that results in death.”19 Many errors do not lead to patient harm, and harm may not be associated with a specific er-ror.26 Focusing on errors shifts the discussion toward individual blame, whereas concentrating on events experienced by patients helps to keep the focus on systemic improvement to reduce patients’ suffering.
Our work on the dental clinic trigger tool re-minds us that improving quality and safety is a journey rather than a destination to be reached in one fell swoop. This first step was limited to two practice settings: the study site’s TPs and FP, which cover the full gamut of training levels (predoctoral students to specialist faculty mem-bers) and specialties. The implant failure trigger, for instance, may be less predictive of AEs in clin-ics in which practitioners do not place implants, and the performance of the dental clinic trigger tool in its entirety may vary according to site.
It is likely that expanding the dental clinic trigger tool to include additional triggers (such as letters of complaint) will make it more effec-tive and robust to differences across sites. As the trigger tool evolves, so will the range of AEs
ously measured. Health Aff (Millwood) 2011;30(4):581-589. 19. Institute for Healthcare Improvement, Resar R. Outpatient Ad-
verse Event Trigger Tool: 2006. www.ihi.org/knowledge/Pages/Tools/OutpatientAdverseEventTriggerTool.aspx. Accessed May 31, 2013.
20. Kalenderian E, Ramoni RL, White JM, et al. The development of a dental diagnostic terminology. J Dent Educ 2011;75(1):68-76.
21. American Dental Association. CDT 2013: Dental Procedure Codes. Chicago: American Dental Association; 2012.
22. National Coordinating Council for Medication Error Reporting and Prevention. NCC MERP index for categorizing medication er-rors. Rockville: National Coordinating Council for Medication Error Reporting and Prevention; 2001.
23. Ramoni RB, Walji MF, White JM, et al. From good to better: toward a patient safety initiative in dentistry. JADA 2012;143(9): 956-960.
24. Commission on Dental Accreditation, American Dental Asso-ciation. Accreditation standards for dental education programs. Chicago: American Dental Association; 2007. www.ada.org/sections/educationAndCareers/pdfs/predoc_2013.pdf. Accessed June 14, 2013.
25. Institute for Healthcare Improvement. Introduction to trigger tools for identifying adverse events. www.ihi.org/knowledge/Pages/Tools/IntrotoTriggerToolsforIdentifyingAEs.aspx. Accessed June 3, 2013.
26. Layde PM, Cortes LM, Teret SP, et al. Patient safety efforts should focus on medical injuries. JAMA 2002;287(15):1993-1997.
11. Lee JS, Curley AW, Smith RA; Institute of Medicine. Prevention of wrong-site tooth extraction: clinical guidelines. J Oral Maxillofac Surg 2007;65(9):1793-1799.
12. Committee on Quality of Health Care in America, Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington: National Academy Press; 2001.
13. Leape LL, Kabcenell AI, Gandhi TK, Carver P, Nolant W, Berwick DM. Reducing adverse drug events: lessons from a break-through series collaborative. Jt Comm J Qual Improv 2000;26(6): 321-331.
14. Jha AK, Kuperman GJ, Teich JM, et al. Identifying adverse drug events: development of a computer-based monitor and compari-son with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998;5(3):305-314.
15. Resar RK, Rozich JD, Classen D. Methodology and rationale for the measurement of harm with trigger tools. Qual Saf Health Care 2003;12 suppl 2:ii39-ii45.
16. Griffin F, Resar R. IHI Global Trigger Tool for Measuring Adverse Events. 2nd ed. Cambridge, Mass.: Institute for Healthcare Improvement; 2009. www.ihi.org/knowledge/Pages/IHIWhitePapers/IHIGlobalTriggerToolWhitePaper.aspx. Accessed May 29, 2013.
17. Resar RK, Rozich JD, Simmonds T, Haraden CR. A trigger tool to identify adverse events in the intensive care unit. Jt Comm J Qual Patient Saf 2006;32(10):585-590.
18. Classen DC, Resar R, Griffin F, et al. ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previ-
![Untitled-1 [chitrapurmath.net] · 2019-11-22 · 4m pm 8.30 - pm 11:00 am 11.30 am 5m - 6:00 pm 6.30 pm 8.30 - am 10:00 am 11:00 am 11.30 am 5.45 pm 7:00 - 8:00 pm pm SPECIAL PROGRAMME](https://static.documents.pub/doc/80x56/5f3d806568ad817ada3e5ec7/untitled-1-2019-11-22-4m-pm-830-pm-1100-am-1130-am-5m-600-pm-630.jpg)
