2017-01-23
1
Assessment Assessment
MethodologiesMethodologiesWorkshopWorkshop
Pro Board Annual ConferencePro Board Annual Conference
San Diego, January 20/21, 2017San Diego, January 20/21, 2017
Assessment MethodologiesAssessment Methodologies
Presenters:Presenters:
• Fred Piechota
• Brian Brauer
• Chris Senaratne
• Craig Richardson
Assessment MethodologiesAssessment Methodologies
Objectives:Objectives:
• Overview of the Pro Board assessment
methodologies
• Discuss some examples of each
• Application for change in accredited process
• Group dialogue on project methodology
direction
• Q&A
Assessment MethodologiesAssessment Methodologies Applicable CriteriaApplicable Criteria
TD2:TD2:
• The agency shall have the ability to completely
test the level(s) of each applicable standard
with justifiable test methodologies.
2017-01-23
2
Applicable CriteriaApplicable Criteria
TD3:TD3:
• Procedures shall be in place in the test
development, administration, review, and
improvement processes to ensure the
production of valid and reliable test
instruments and test results.
Applicable CriteriaApplicable Criteria
TA5:TA5:
• Procedures shall be in place to ensure that skills tests are assessed by qualified persons who are not involved in the training of the candidate in the skill being tested; that qualified persons administer (proctor) cognitive tests; and that qualified persons evaluate projects, portfolio submissions, or other methodologies (if used).
Applicable CriteriaApplicable Criteria
TA6:TA6:
• Procedures shall be in place to ensure that
skills tests, written exams, projects, portfolio
submissions, or other evaluation
methodologies are administered in a
consistent and impartial manner.
Written CognitiveWritten Cognitive Written CognitiveWritten Cognitive
Definition:Definition:
• The cognitive (knowledge) assessment methodology is used to assess a candidate’s subject matter knowledge by means of a written or oral test in which the candidate is required to provide specific answers to specific questions related to the JPRs of the standard/level to which the candidate is seeking certification.
2017-01-23
3
Written CognitiveWritten Cognitive
• Assessment made through a Test Instrument –
a set of questions used to evaluate the
candidate
• Test Instrument is made up of a multiple
number of Test Items (individual questions)
from a Test Bank
Written CognitiveWritten Cognitive
• The test bank must include test items that are designed to assess, and correlated to, each of the areas of the standard that are going to be tested cognitively.
• Random selection from that correlated bank of test items can be used to generate a test instrument.
• Another methodology of selection is stratified or targeted random selection, where the random selection is done from the subsets of test items for each section of the standard for which cognitive evaluations are used.
Written CognitiveWritten Cognitive
• Test Banks can be developed locally or
purchased
• Development processes of NFPA, PTS, IFSTA,
Jones & Bartlett have been approved by
ProBoard
• Local validation by subject matter experts is
necessary for both locally developed or
purchased test banks
Psychomotor SkillsPsychomotor Skills
Psychomotor SkillsPsychomotor Skills
Definition:Definition:
• The psychomotor (skills) assessment methodology is used to assess a candidate’s ability to perform a physical task by means of a real-time skills performance evaluation. The candidate is required to correctly perform the physical task/skill identified by the JPR and is evaluated on his/her performance in real-time as the candidate performs the task.
PortfolioPortfolio
2017-01-23
4
PortfolioPortfolio
Definition:Definition:
• The portfolio method of evaluating personnel for certification may be utilized to evaluate job performance requirements, requisite knowledge, requisite skills, and objectives of standards when other methods of evaluation are not applicable, or when the accredited agency identifies that the portfolio method is an appropriate method of evaluation on the standard-to-test correlation matrix. The portfolio process is intended to measure the candidate’s accomplishments with respect to specific job performance requirements (JPR) or objectives within the various levels of certification that are part of the National Professional Qualifications System.
PortfolioPortfolio
Definition:Definition:
• The portfolio method of evaluation shall not
be confused with other processes used to
identify or verify that a candidate has met a
single job performance requirement or
objective, such as letters of proof from a
supervisor, certificates, transcripts, etc.
PortfolioPortfolio
Definition:Definition:
• The Portfolio process is based on the
evaluation of a candidate’s experience,
training, and education to ensure compliance
with the job performance requirements or
objectives of the appropriate standard.
PortfolioPortfolio
• Different from assessing pre-requisites or establishing an entry point into the certification system
– because certification can be granted upon completion
• Must be a documented, valid and reliable process that correlates the candidate’s submitted accomplishments to the JPRs in the level to which certification is sought
Other MethodologiesOther Methodologies
Definition:Definition:
• The other assessment methodology column in
the Assessment Methodology Matrix is used
to identify a valid and reliable assessment tool
for all or part of a certification assessment
that is not described above.
ProjectProject
2017-01-23
5
ProjectProject
Definition:Definition:
• The project-based assessment (PBA) method of evaluating personnel for certification may be utilized to evaluate job performance requirements, requisite knowledge, requisite skills, and objectives of a standard. The instrument used to evaluate the certification candidate using the PBA method is known as a project.
ProjectProject
Definition:Definition:
• A project is an assignment that requires the application of knowledge to perform a task that is not assessed/evaluated as it is being performed. The completion of the task must include a requirement for the candidate to submit an output product that can be assessed by a project evaluator to determine success of the candidate.
ProjectProject
Definition:Definition:
• An assessment is likely a project if: – the evaluator is not required to witness the application of the
knowledge for the task taking place (e.g., skill set to develop a budget cannot be witnessed by an evaluator), or
– the assessment has a “take home” component or a component that is completed without direct supervision of a proctor or evaluator (e.g., receive an assignment during class and have to take it home to complete it), or
– the assessment has several plausible outcomes and the evaluation tools must be flexible enough to allow for that (e.g., develop a capital improvement plan that could be accomplished several correct ways), or
– has 6 months to complete a task and submits a product for final evaluation)
ProjectProject
Definition:Definition:
• Typically, a PBA is an evaluation of a product (such as a budget, long range plans, reports, or letters) rather than a process (such as donning, tying knots, raising ladders, or decontamination). If the evaluator has to watch someone complete each individual step, it is likely a psychomotor skill; however, if an evaluator does not have to witness each step but rather evaluates a completed product, it is likely a PBA.
ProjectProject
Definition:Definition:
• The Self-Assessment has several criteria that must be addressed if an agency elects to use PBA. There must be a process for evaluating assigned projects, as found in the Test Administration and Test Development sections of the Self-Assessment, that accurately measure compliance with a specific job performance requirement, requisite knowledge, requisite skill or objective. It is preferred that the instructor is not the evaluator; however, where an agency has established an approved evaluation tool to score the project (i.e. a rubric, scoring criteria, or model example used by the evaluator to assess all the candidates in a consistent manner), it may be acceptable that the instructor evaluates the candidates. This tool must have been developed in accordance with the agencies policies as defined in the Test Development section of the Self-Assessment.
Change in Accredited ProcessChange in Accredited Process
2017-01-23
6
Change in Accredited ProcessChange in Accredited Process
• Similar to the application to extend to a new standard or level
• Agencies are accredited based upon the processes submitted in the application and self assessment document; and at a site visit
• Your work is dynamic and your business model often changes to remain responsive to constituents
• New process must be approved
Change in Accredited ProcessChange in Accredited Process
Examples:Examples:
• Changing test development procedures (i.e. switching test bank providers or beginning to develop test questions in-house)
• Beginning the use of a new assessment methodology
• Introducing remote testing when not previously approved
• Online testing
Change in Accredited ProcessChange in Accredited Process
How it works:How it works:
• Complete the application form
• Assess the new process for how it differs from existing approved processes and identify the criteria that relate to the new process
• Answer the appropriate sections of the self assessment
• Provide a detailed summary of the new process and how it is administered
ProjectProject
2016 Conference Information:2016 Conference Information:
• Compliance with definition upon any of the
following occurring after July 1, 2016:
– Next scheduled site visit
– Application to extend to new levels
– NFPA Standard new edition
ProjectProject
Where we are now:Where we are now:
• COA meeting recap
• Some gaps to address
• Engage agencies in establishing a process to
move forward
• Testing JPR vs Testing RK & RS
COA Opinion 18COA Opinion 18
• The Committee on Accreditation (COA) has ruled that when completing a Standard to Test Correlation Matrix, the completion of either the Job Performance Requirement (JPR) section of the matrix or the Requisite Knowledge (RK) and the Requisite Skill (RS) sections of the matrix meets the COA’s requirements. It follows therefore, that the evaluation process shall measure either the JPR or the RK and RS for each objective in the standard.
2017-01-23
7
NFPA 1033 Annex B
• B.3.1 Certification. JPRs can be used to
establish the evaluation criteria for
certification at a specific job level. When used
for certification, evaluation must be based on
the successful completion of JPRs.
NFPA 1033 Annex B
• First, the evaluator would verify the
attainment of requisite knowledge and skills
prior to JPR evaluation. This might be through
documentation review or testing.
NFPA 1033 Annex B
• Next, the candidate would be evaluated on
completing the JPRs. The candidate would
perform the task and be evaluated based on
the evaluation parameters and/or
performance outcomes. This performance-
based evaluation can be either practical (for
psychomotor skills such as “ventilate a roof”)
or written (for cognitive skills such as
“interpret fire patterns”).
NFPA 1033 Annex B
• Note that psychomotor skills are those
physical skills that can be demonstrated or
observed. Cognitive skills (or mental skills)
cannot be observed but are evaluated on how
one completes the task (process-oriented) or
on the task outcome (product-oriented).
Which Methodology?Which Methodology? Which Methodology?Which Methodology?
2017-01-23
8
Which Methodology?Which Methodology? Which Methodology?Which Methodology?
Which Methodology?Which Methodology? Which Methodology?Which Methodology?
Which Methodology?Which Methodology?