Date post: | 17-Jan-2016 |
Category: |
Documents |
Upload: | christiana-elisabeth-wilkerson |
View: | 219 times |
Download: | 0 times |
Russ Ryan, Vice PresidentNational Biometric Security Project
The Importance ofThe Importance of Biometric Testing Biometric Testing
Biometrics for National Security (BiNS)
National Signatures Project National Energy
Technology Lab (NETL) NIST International Organization for Migration (IOM) Office of Presidential Affairs (UAE) International Labour Organization (ILO) BioAPI Consortium State of West Virginia
NBSP
National Biometric Security ProjectNational Biometric Security Project
Biometric ApplicationsBiometric Applications
HSPD-24
GAO
?Robust biometric passports
Financial and medical services authorizations
Border and travel services
Drivers’ licenses
Physical and Logical access
Increasing reliance on biometrics to secure access, transactions & identity
Equally increasing demand for accurate, unbiased evaluations
Testing can provide accurate metrics on how the technology will perform in the real world
Alleviating unfounded concerns about operational performance
Understanding Biometric PerformanceUnderstanding Biometric Performance
Universal
Unique
Permanence
Collectable
Performance
Acceptance
Spoof Resistance
Attributes of an Ideal BiometricAttributes of an Ideal Biometric
Biometric Testing TodayBiometric Testing Today
Performance of biometric systems is a function of:
strength of the underlying biometric. quality and information content of the input configuration and architecture of the system the relationship of accuracy and throughput error rates, the nature of failures and their cost, and system vulnerabilities which contribute to an overall assessment of system performance
Increasingly, biometric devices are components of larger systems imposing external variables that impact biometric system performance in
the field
Biometric Testing TodayBiometric Testing Today
Three major considerations in testing biometric products
dependence of measured error rates on the application need for a large test population necessity for a time delay between enrollment and
testing
Comparison of Testing TypesComparison of Testing Types
Technology Testing
Goal: Produce a repeatable and scalable assessment of an algorithm/sensor using offline data processing
Scenario Testing
Goal: Determine overall system performance (both algorithmic & human factors performance measures)
Operational Testing
Goal: Determine biometric system performance in a specific environment with a specific target population
* Best Practices in Testing and Reporting Performance of Biometric Devices, by A. J. Mansfield, National Physical Laboratory and J. L. Wayman, San Jose State University. Published 2002 by The Centre for Mathematics and Scientific Computing ,National Physical Laboratory, Queens Road, 88, Middlesex, England.
Technology TestingTechnology Testing
Understand/compare software techniques used to acquire, process and compare biometric data
Main focus is on the pattern matching technique used to compare biometric data
Evaluates different classification and matching methods on efficiency, speed and performance
Offline processing of data carried out in laboratory Evaluation compares competing algorithms
from a single type of technology carried out on a standardized database collected by a universal sensor results determine the relative effectiveness of the tested algorithms
Scenario TestingScenario Testing
Evaluates performance across biometric devices
Each system has its own acquisition sensor and receives different data inputs than those tested in technology (algorithm) evaluation
Data collected for all tested systems must come from same environment and same population
Test results are only considered repeatable under identical control variables & environment
Scenario evaluation helps an end user decide which biometric device has the potential to work best for his/her needs
Operational TestingOperational Testing
Determine performance of abiometric system in a real application environment
Population and environment are not controlled
System vulnerability can also be performed
Helps determine how system as a whole will
perform by testing a live system in its native
environment for its intended application
Conformance TestingConformance Testing
Determines conformance with relevant published
ISO/IEC standards Utilizes conformance test
suites designed for specific standards
Evaluations will expand to include additional
standards as the software modules are written
and field tested
Standards Evaluated
Target Value
INCITS 377-2004 Pass/Fail
INCITS 378-2004 Pass/Fail
ISO 19794-2-2005 Pass/Fail
INCITS 379-2004 Pass/Fail
INCITS 381-2004 Pass/Fail
INCITS 385-2004 Pass/Fail
ISO 19794-2-2005 Pass/Fail
INCITS 396-2005 Pass/Fail
INCITS 395-2005 Pass/Fail
ILO SID Pass/Fail
ICAO LDS 1.7 Pass/Fail
BioAPI Pass/Fail
Vulnerability TestingVulnerability Testing
Impersonation attempts
(disguises) or spoofing
(artifact substitution for live
feature)Database attacks (exchanging
or corrupting references)Tampering with threshold
settings Network-based attacks
Product “vulnerabilities” must be defined in the context of the operating environment and proper usage within the
design parameters of the product
Interoperability TestingInteroperability Testing
Multi-modal systems demand acceleration of biometric interoperability
Interoperability testing assesses
ability to exchange and use information on a single system in a multi-modal environment
interface of the biometric component with the holistic security program
Interoperability Trade-offs…Interoperability Trade-offs…
Lowers complexity of the application- Re-use- Future Proofing- Vendor independence- Upgrade path- Simplifies CM- Simplified integration
-Product optimization- Better performance
-Lower level control- More sophistication
-Can be faster to market (due to standards development time)
May incur additional overhead- May not be able to take advantage of vendor unique capabilities- Interfaces are generic and consensus based, so may not be optimized for a particular use
-Custom interfaces for each proprietary product to be interfaced
- Increased cost/complexity- Added CM
- Product changes affect application- Can result in vendor dependence
Standard Proprietary
Ad
van
tag
es
Dis
ad
van
tag
es
Courtesy of Cathy Tilton, VP Standards & Technology, Daon
Usability TestingUsability Testing
Intuitiveness of the system interface with the user community
Is the transaction an inviting and positive experience?
Is consistent instruction and feedback built into the process?
Is the performance reliable for operational staff as well as users?
Qualified Product List TestingQualified Product List Testing
First initiated and commercialized by NBSP
Utilizes comprehensive scenario test capability
Initially used to identify products that successfully passed common performance thresholds
Increasingly tailored to the application
QPL Testing BenefitsQPL Testing Benefits
Catalog of commercially available products that meets minimum standards for a specific application
Significant reduction in duplicative pilot tests
Acceleration of acquisition process by identifying a field of suitable products
Opportunity for vendors’ to demonstrate general or specified performance capabilities
Factors Affecting Biometric PerformanceFactors Affecting Biometric Performance
Variations in: biometric pattern the way users present the
biometric the way the sensor reads the
biometric System scalability the transmission process
(including noise introduced by compression & expansion)
User acceptance/application-specific limitations
Additional Measurement ParametersAdditional Measurement Parameters
Reliability, availability, scalability, maintainability Security, including vulnerability to spoofing Human factors, including user acceptance Cost/benefit in comparison to existing security
processes and systems Privacy regulation compliance
Laboratory CertificationLaboratory Certification
BSI awarded ISO/IEC 17025 Accreditation specifies requirements for competency to conduct
biometric tests
covers testing performed using standard methods, non-standard methods and laboratory-developed method
laboratory customers, regulatory authorities and accreditation bodies use it to confirm the competency of laboratories.
NISTNIST Handbook 150-25 with technical requirements and
guidance for accreditation of laboratories under the NVLAP Biometrics Testing program released Sept. 2009
Russ Ryan, [email protected]
www.nationalbiometric.org
The Importance ofThe Importance of Biometric Testing Biometric Testing