Russ Ryan, Vice PresidentNational Biometric Security Project

The Importance ofThe Importance of Biometric Testing Biometric Testing

Biometrics for National Security (BiNS)

National Signatures Project National Energy

Technology Lab (NETL) NIST International Organization for Migration (IOM) Office of Presidential Affairs (UAE) International Labour Organization (ILO) BioAPI Consortium State of West Virginia

NBSP

National Biometric Security ProjectNational Biometric Security Project

Biometric ApplicationsBiometric Applications

HSPD-24

GAO

?Robust biometric passports

Financial and medical services authorizations

Border and travel services

Drivers’ licenses

Physical and Logical access

Increasing reliance on biometrics to secure access, transactions & identity

Equally increasing demand for accurate, unbiased evaluations

Testing can provide accurate metrics on how the technology will perform in the real world

Alleviating unfounded concerns about operational performance

Understanding Biometric PerformanceUnderstanding Biometric Performance

Universal

Unique

Permanence

Collectable

Performance

Acceptance

Spoof Resistance

Attributes of an Ideal BiometricAttributes of an Ideal Biometric

Biometric Testing TodayBiometric Testing Today

Performance of biometric systems is a function of:

strength of the underlying biometric. quality and information content of the input configuration and architecture of the system the relationship of accuracy and throughput error rates, the nature of failures and their cost, and system vulnerabilities which contribute to an overall assessment of system performance

Increasingly, biometric devices are components of larger systems imposing external variables that impact biometric system performance in

the field

Biometric Testing TodayBiometric Testing Today

Three major considerations in testing biometric products

dependence of measured error rates on the application need for a large test population necessity for a time delay between enrollment and

testing

Comparison of Testing TypesComparison of Testing Types

Technology Testing

Goal: Produce a repeatable and scalable assessment of an algorithm/sensor using offline data processing

Scenario Testing

Goal: Determine overall system performance (both algorithmic & human factors performance measures)

Operational Testing

Goal: Determine biometric system performance in a specific environment with a specific target population

* Best Practices in Testing and Reporting Performance of Biometric Devices, by A. J. Mansfield, National Physical Laboratory and J. L. Wayman, San Jose State University. Published 2002 by The Centre for Mathematics and Scientific Computing ,National Physical Laboratory, Queens Road, 88, Middlesex, England.

Technology TestingTechnology Testing

Understand/compare software techniques used to acquire, process and compare biometric data

Main focus is on the pattern matching technique used to compare biometric data

Evaluates different classification and matching methods on efficiency, speed and performance

Offline processing of data carried out in laboratory Evaluation compares competing algorithms

from a single type of technology carried out on a standardized database collected by a universal sensor results determine the relative effectiveness of the tested algorithms

Scenario TestingScenario Testing

Evaluates performance across biometric devices

Each system has its own acquisition sensor and receives different data inputs than those tested in technology (algorithm) evaluation

Data collected for all tested systems must come from same environment and same population

Test results are only considered repeatable under identical control variables & environment

Scenario evaluation helps an end user decide which biometric device has the potential to work best for his/her needs

Operational TestingOperational Testing

Determine performance of abiometric system in a real application environment

Population and environment are not controlled

System vulnerability can also be performed

Helps determine how system as a whole will

perform by testing a live system in its native

environment for its intended application

Conformance TestingConformance Testing

Determines conformance with relevant published

ISO/IEC standards Utilizes conformance test

suites designed for specific standards

Evaluations will expand to include additional

standards as the software modules are written

and field tested

Standards Evaluated

Target Value

INCITS 377-2004 Pass/Fail

INCITS 378-2004 Pass/Fail

ISO 19794-2-2005 Pass/Fail

INCITS 379-2004 Pass/Fail

INCITS 381-2004 Pass/Fail

INCITS 385-2004 Pass/Fail

ISO 19794-2-2005 Pass/Fail

INCITS 396-2005 Pass/Fail

INCITS 395-2005 Pass/Fail

ILO SID Pass/Fail

ICAO LDS 1.7 Pass/Fail

BioAPI Pass/Fail

Vulnerability TestingVulnerability Testing

Impersonation attempts

(disguises) or spoofing

(artifact substitution for live

feature)Database attacks (exchanging

or corrupting references)Tampering with threshold

settings Network-based attacks

Product “vulnerabilities” must be defined in the context of the operating environment and proper usage within the

design parameters of the product

Interoperability TestingInteroperability Testing

Multi-modal systems demand acceleration of biometric interoperability

Interoperability testing assesses

ability to exchange and use information on a single system in a multi-modal environment

interface of the biometric component with the holistic security program

Interoperability Trade-offs…Interoperability Trade-offs…

Lowers complexity of the application-   Re-use-   Future Proofing-   Vendor independence-   Upgrade path-   Simplifies CM-   Simplified integration

-Product optimization- Better performance

-Lower level control- More sophistication

-Can be faster to market (due to standards development time)

May incur additional overhead-   May not be able to take advantage of vendor unique capabilities-   Interfaces are generic and consensus based, so may not be optimized for a particular use

-Custom interfaces for each proprietary product to be interfaced

- Increased cost/complexity- Added CM

-  Product changes affect application-  Can result in vendor dependence

Standard Proprietary

Ad

van

tag

es

Dis

ad

van

tag

es

Courtesy of Cathy Tilton, VP Standards & Technology, Daon

Usability TestingUsability Testing

Intuitiveness of the system interface with the user community

Is the transaction an inviting and positive experience?

Is consistent instruction and feedback built into the process?

Is the performance reliable for operational staff as well as users?

Qualified Product List TestingQualified Product List Testing

First initiated and commercialized by NBSP

Utilizes comprehensive scenario test capability

Initially used to identify products that successfully passed common performance thresholds

Increasingly tailored to the application

QPL Testing BenefitsQPL Testing Benefits

Catalog of commercially available products that meets minimum standards for a specific application

Significant reduction in duplicative pilot tests

Acceleration of acquisition process by identifying a field of suitable products

Opportunity for vendors’ to demonstrate general or specified performance capabilities

Factors Affecting Biometric PerformanceFactors Affecting Biometric Performance

Variations in: biometric pattern the way users present the

biometric the way the sensor reads the

biometric System scalability the transmission process

(including noise introduced by compression & expansion)

User acceptance/application-specific limitations

Additional Measurement ParametersAdditional Measurement Parameters

Reliability, availability, scalability, maintainability Security, including vulnerability to spoofing Human factors, including user acceptance Cost/benefit in comparison to existing security

processes and systems Privacy regulation compliance

Laboratory CertificationLaboratory Certification

BSI awarded ISO/IEC 17025 Accreditation specifies requirements for competency to conduct

biometric tests

covers testing performed using standard methods, non-standard methods and laboratory-developed method

laboratory customers, regulatory authorities and accreditation bodies use it to confirm the competency of laboratories.

NISTNIST Handbook 150-25 with technical requirements and

guidance for accreditation of laboratories under the NVLAP Biometrics Testing program released Sept. 2009

Russ Ryan, rryan@nationalbiometric.org703-201-8179

www.nationalbiometric.org

The Importance ofThe Importance of Biometric Testing Biometric Testing