47
#EATPconf Managing the Rapid Pace of Change: An Insider’s View Session Presented at EATP in Lisbon, Portugal September 29, 2016
#EATPconf
Managing the Rapid Pace of Change: An
Insider’s View
Session Presented at EATP in Lisbon, PortugalSeptember 29, 2016
#EATPconf
Presenters
Dania Eter, MBADirector, Global Certifications,
Cisco Systems
Liberty Munson, PhDPrincipal Psychometrician & Assessment
Quality Lead, Microsoft Learning
Experiences
Lynn Anderson, PhDCEO, International Joint Commission on
Allied Health Personnel in Ophthalmology
9/6/16 2
#EATPconf
Managing Change in Exam Development and
Maintenance: Cisco’s Journey Dania Eter, MBA, Director, Global Certifications
Cisco Systems
#EATPconf
Agenda
■ About Cisco
■Challenges
■ Best Practices and Recommendations
9/13/2016 4
About Cisco
9/13/2016 5
#EATPconf
Cisco Certifications
■ Certification program inception in 1993■ Certified over 2.8 million people■ More than 200 exams offered ■ 5,000+ locations in 182 countries■ Multiple languages■ Career Certification Levels
6
VisionCreate the next generation of the workforce that will enable the digital
transformation across the industry
Entry Associate Professional Expert Architect
#EATPconf
Team Charter
■ Design, develop, deliver, and maintain Cisco certification exams
■ Develop exams that are psychometrically sound, reliable, valid, and legally defensible
■ Adhere to testing industry standards and comply with ISO/ANSI requirements
■ Implement innovative ways to enhance exams while protecting exam integrity
■ Implement legal protections and security processes to protect exam content and Cisco IP
■ Research industry trends and new assessment models to grow programs
#EATPconf 8
Job Analysis
ValidationStudies
Cut ScorePanel
PreliminaryCandidate Pool
DefinePractice
ExamBlueprint
Compose Exam Set Passing Score
ImplementExam
Issue Candidate Results
Develop Items (See Details)
ITEMBANK
MeasurePerformance
DraftQuestions
Technical Edit
Psycho-metric Edit
Grammar EditPractitioner
Review
Pretest
Source: Exceptional Certification, Roger L. Brauer
Exam Development Lifecycle
Challenges
9/13/2016 9
#EATPconf
Expansive Certification Portfolio
Growth Rate over 2 years
Include all certifications and specialist certs
Data as of July FY16
Americas 924,266 (+26%)
(+16%)
(+9%)
(+77%)
(+61%)
(+21%)
(N/A)
(+24%)
EMEAR 772,372 (+27%)
(+17%)
(+13%)
(+76%)
(+64%)
(+22%)
(N/A)
(+16%)
GC 348,657 (+12%)
(+8%)
(+5%)
(+50%)
(+58%)
(+23%)
(N/A)
(+14%)
APJ 812,766 (+17%)
(+13%)
(+9%)
(+84%)
(+67%)
(+17%)
(N/A)
(+15%)
World Wide 2,858,061 (+22%)
(+14%)
(+10%)
(+76%)
(+63%)
(+21%)
(N/A)
(+19%)
#EATPconf
Keeping Up with Changes
9/13/2016 11
New Technologies
• Constant evaluation
• Quarterly validation and
updates
Mature Technologies
• JTA every 2-3 years
• Mid-cycle practice analysis
#EATPconf
SME Engagement
■ Finding qualified volunteer experts, especially with new and evolving technologies
■Consistent engagement during projects to revise/update exams
#EATPconf
SME Recruitment and Retention
Recruitment■ Online SME Recruitment
Portal
■ Internal contacts
■ Cisco events
■ Technology conferences
■ Contract SMEs through vendors
Incentives■ Thank You note to manager
■ Recertification extension
■ Cisco Press books
■ Voucher for written exam
9/13/2016 13
#EATPconf
Security
Types of Threats
Candidate Cheating
Corrupt Test Centers
Flaws in the Delivery System
Internal Leaks
Organized Test Theft
Ring Syndicates
Weak Legal Protections
Cultural Influencers
Systems & Tools
Internal Processes
#EATPconf
Candidate enforcements
Mitigation Strategies
Controlling exam exposure with targeted, multi-layered approach
Continual exam content development – items & multiple forms
Web-monitoring for exam exposure detection and investigations
Data analysis resulting in closed sites
Layer 6
Layer 5
Layer 4
Layer 3
Layer 2
Layer 1
Special initiatives detecting the source of exam breaches
Exploring alternative delivery models - IBT Testing
#EATPconf
The Changing Landscape
New mission: testing software developers
#EATPconf
Agility in Design and Development
■ Increase design agility to reflect new technologies for currency and accuracy
■ Agile, flexible exam development methodologies
■ SMEs involved early and often to determine necessary changes
■ Item types that effectively test coding skills
■Modular testing
■ Evolving portfolio to accommodate next generation of assessment programs
#EATPconf
Collaborate with Industry Partners
9/13/2016 18
■ Thinking beyond test delivery
■ Position industry best practices for IT testing lifecycle
■ Evaluate and adopt services that meet the needs of the current market while supporting and advancing our global community
Business
Development
Pioneer
New
Practices
Customer
Focused
ApproachText
#EATPconf
Best Practices and Recommendations
■ Build strong relationship with SMEs, especially in new or changing technology areas
■Work closely with vendors and partners to collaborate on solutions
■ Develop agile processes to allow for faster updates to exam content
■Constant content development
■ Review and respond to customer comments
#EATPconf
Managing Change in Exam Development and
Maintenance: Microsoft’s Journey Liberty J. Munson, PhD, Principal Psychometrician
Microsoft Learning Experiences
#EATPconf
Enabling the next generation of innovators, entrepreneurs and
developers
Enable Customers
& Partners at Scale
Learning Credits
Badges
Microsoft Professional Degree
Machine Learning
Microsoft Azure Fundamentals
Programming in C#
MOC On Demand
Creative Coding through Games and Apps
Harvard/MSFT CS50.AP
C# for Absolute Beginners
Hour of Code
Minecraft
Drive Usage, Deployment& Consumption
Empower Students& Educators
Activate Cloud &
Productivity Partners
2
5
Adoption
Drive Deep Engagement
Advocacy
Enable Capability Mastery
Affinity
Spark Excitement
Total reach
New/Competitive Developers reached
Skilled technologists building on Microsoft platforms and services
Preference for Microsoft platforms & services
1 3 4
#EATPconf
An Overview of MS Certification Exams
■Computer administered
■ Test center deliveries or online proctoring
■Global distribution
■Ongoing delivery
■ Variety of item types (e.g., multiple choice, drag and drop, active screen, hot area, case studies, labs, text entry, code analysis, etc.)
9/13/2016 22
#EATPconf
Microsoft Learning Experiences
Drive adoption of Microsoft technologies through training and certification
Similar paths for Office:
• Microsoft Office Specialist
• Microsoft Office Expert
• Microsoft Office Master
#EATPconf
Exam Development Process
Objective
DomainBlueprint
Item Development
Research
Candidate
Manager
Alpha Review &Item Revision
Field Testing (a.k.a. Beta Exam)
Forms Assembly
LocalizedExamStandard
Setting
Item Selection
Publish Exam
Ongoing Psychometrics
#EATPconf
Mitigation efforts include:■ Test blueprint focuses on KSAs less likely to become
obsolete SMEs rate objectives on importance for success, frequency, and
potential for obsolescence
■ Item writing focuses on fundamental concepts rather than specifications Fundamental concepts are more stable
Function, purpose, application/use, intent
Minimize application or version specific content
Must resist being on the cutting edge in this context
Exam Development and Item Writing Strategies
#EATPconf
• Ensure competence in use of Microsoft products• Drive adoption and sales• Certify as many people as possible as soon as our
technologies become available
• Exam in market around the same time as general availability of the software
• Development begins on “vaporware”
• Features and functionality of early versions are often in flux
• Features may not be available but the product group plans to include them in the final release, and we may need to develop items for them
#EATPconf
Designing & Developing Exams on Vaporware
■Close partnership with business group
■ Identifying “right” SMEs Have early access to software
■Changes communicated as early as possible “Feature complete” version shared with SMEs who “scrub” content
domain and items to ensure relevance and technical accuracy
■ Beta candidate comments Comments about technical accuracy, clarity, relevance, etc.
reviewed with SMEs
Occasionally comment on the long term viability of items that may become obsolete over time if rumored changes are implemented
■One final review Items are scrubbed before publication to ensure accuracy and
relevance
#EATPconf
• As Microsoft migrates more software to the cloud and leverages its flexibility, move away from major version releases of our software • Azure, Office 365
• Changes and improvements will be “evolutions” (small but constant change) rather than “revolutions” (big changes that are usually the hallmarks of the next version/edition), which historically drove exam refresh projects
• Changes happen quickly and seamlessly• Can be minor (e.g., feature names) to more
substantial (e.g., user interface, functionality)• Any change could effect exam content, skills assessed,
and the validity of our certification process
#EATPconf
Keeping up with Pace of Change
Build strong partnerships with
BG
Monthly or quarterly review
of content domain
•Did anything change that would affect exam?
What is the change?
•Name change = find & replace?
•Functionality change Review items
associated with anything that
changed
•Are the actual items affected by change?
Can items be fixed or must
they be removed?
Create items to replace those
that are removed
#EATPconf
Best Practices and Recommendations
■ Build strong partnerships with those who have early insight into changes
■ Plan regular reviews and updates
■Ongoing content development Retainer model
■ Listen to test-takers
■ Are exams the best way to measure these skills? What might be a better approach?
#EATPconf
Managing Change in Exam Development and
Maintenance: Cisco’s Journey
Dania Eter, MBA, Director, Global CertificationsCisco Systems
#EATPconf
Managing Change in Exam Development and
Maintenance: IJCAHPO’s Journey
Lynn Anderson, PhD, CEOInternational Joint Commission on Allied Health Personnel in Ophthalmology
#EATPconf
Key Discussions
IJCAHPO Overview
Leadership Challenges
Technology Challenges
Lessons Learned
Best Practices
9/6/16
#EATPconf
IJCAHPO Overview
Certification and licensure programs
Ophthalmology - Inception in 1969
Over 30,000 eye care
professionals globally
Eleven exams offered
MCQ Examinations
Computer-based Simulations
#EATPconf
Leadership Challenges
9/6/16
Top Leadership
Internal Staff Support
SMEs
Key Strategies
Research
Test development models
Scoring system models
Standardization of testing
and evaluation
Greater test reliability
Evaluation bias reduced
ROI demonstrated
Long-term cost savings
Litigation risks reduced
Leadership Support
#EATPconf
Computer-Based Test Publishing
9/6/16
Vendor Support Resources Required
Computer testing vendor
with system capabilities
to publish, administer and
deliver
Test driver
Test delivery software
Web-based technologies
Modular design provides
custom solutions
Navigation
Scoring
Presentation
Item types – including
simulations
#EATPconf
Testing Technology and Trends
Observation Computer
We’ve come a long way
Virtual RealityPaper
#EATPconf
Development Challenges
Research
Relevant to practice: proper task or case
development
Developing reliable and
valid scoring systems
Expert judgment
Financial resources
Psychometric analyses
38
#EATPconf
Agile Development Using
New Technologies
Fidelity and Validity
Accurate presentation of the skill
Procedures presented correctly and incorrectly
Accuracy in measurement - scoring system
Computer programming issues
Navigation (Interface Design)
Validation of all aspects
Tutorial training
Basic technology
Easier to use Authoring tools
Modular Design
#EATPconf
Lessons Learned
Reasonableness of using new technologies Decisions regarding required fidelity Agreement on the criteria for judging a skill Validate repeatedly Test development & programming expertise Candidate preparation needs Public relations Cost and effectiveness of development Dedicated team: leadership, staff, developer,
psychometrician, test administrator
#EATPconf
Best Practices and Recommendations
Leadership engagement
Superiors, team, vendors
Communications with stakeholders: two-way
Frequent examination reviews and monitoring
Continual content development
Trend analysis:
Measurement methods
Data reporting
#EATPconf
Technologies for Assessment
Nuclear Plant SimulationFlight Simulation
Training Simulation
#EATPconf
Types of Testing Technology
■ Diagnostic■ Formative■ Summative■ Adaptive■ Observation■ Orals■ Text-based■ Video■ Simulations■ Virtual Reality■ Haptic
#EATPconf
Future Trends in Assessment
The FutureVirtual Reality, Haptic, Gaming Real
Time Technology, Neural Networks and
Wireless Communications
2015Interactivity sweeps the media center,
Apps, emerging technology
2000Video, Simulations, Big Data and
Learning Bytes
Pre-2000’sPre Computer Testing
#EATPconf
Future Trends in eLearning
Gamification and Simulations
Personal Learning Agents
Five future trends in elearning1.Mobile Devices will become the
platform of choice.2.Augmented Reality will play an
important role in elearning.3.LMS will evolve to support to
measure ability for the performance of users.
4.Gamification and simulations will become an integral part of all elearning experiences.
5.Emergence of Personal Learning Agents to monitor content and provide up-to-date information.
MobileDelivery
Augmented Reality
#EATPconf
Insights for the Future of Assessment
using New Technologies
Using new technologies and modalities of
assessments should be carefully evaluated.
Dressing up in well-polished graphics and
using gaming elements does not substitute
for the sound foundation for scoring
algorithms of achievements.
Using new technologies in assessments does
not automatically add value.
Keep focused on the larger goal of
education and assessment!
#EATPconf
We Can See the Future Now
Thank You!
