REPORT TO THE PRESIDENT Forensic Science in · PDF fileDRAFT – PREDECISIONAL – DO...

DRAFT–PREDECISIONAL–DONOTQUOTEORDISTRIBUTE

1

REPORTTOTHEPRESIDENTForensicScienceinCriminalCourts:

EnsuringScientificValidityofFeature-ComparisonMethods

ExecutiveOfficeofthePresidentPresident’sCouncilofAdvisorson

ScienceandTechnologyVersion101A(August26,2016)

September2016


2


3

REPORTTOTHEPRESIDENT

ForensicScienceinCriminalCourts:EnsuringScientificValidity

ofFeature-ComparisonMethods

ExecutiveOfficeofthePresidentPresident’sCouncilofAdvisorson

ScienceandTechnology

September2016


4

AboutthePresident’sCouncilofAdvisorsonScienceandTechnology

The President’s Council of Advisors on Science and Technology (PCAST) is an advisory group of theNation’s leading scientists and engineers, appointed by the President to augment the science andtechnology advice available to him from inside theWhite House and from cabinet departments andother Federal agencies. PCAST is consulted about, and often makes policy recommendationsconcerning,thefullrangeofissueswhereunderstandingsfromthedomainsofscience,technology,andinnovationbearpotentiallyonthepolicychoicesbeforethePresident.

FormoreinformationaboutPCAST,seewww.whitehouse.gov/ostp/pcast.


v

ThePresident’sCouncilofAdvisorson

ScienceandTechnology

Co-Chairs

JohnP.HoldrenAssistanttothePresidentforScienceandTechnology

Director,OfficeofScienceandTechnologyPolicy

EricS.LanderPresidentBroadInstituteofHarvardandMIT

ViceChairs

WilliamPressRaymerProfessorinComputerScienceandIntegrativeBiology

UniversityofTexasatAustin

MaxineSavitzHoneywell(ret.)

Members

WandaM.AustinPresidentandCEOTheAerospaceCorporation

ChristopherChybaProfessor,AstrophysicalSciencesandInternationalAffairs

PrincetonUniversity

RosinaBierbaumProfessor,SchoolofNaturalResourcesandEnvironment,UniversityofMichigan

RoyF.WestinChairinNaturalEconomics,SchoolofPublicPolicy,UniversityofMaryland

S.JamesGates,Jr.JohnS.TollProfessorofPhysicsDirector,CenterforStringandParticleTheory

UniversityofMaryland,CollegePark

ChristineCasselPlanningDeanKaiserPermanenteSchoolofMedicine

MarkGorenbergManagingMemberZettaVenturePartners


vi

SusanL.GrahamPehongChenDistinguishedProfessorEmeritainElectricalEngineeringandComputerScience

UniversityofCalifornia,Berkeley

EdPenhoetDirectorAltaPartnersProfessorEmeritus,BiochemistryandPublicHealth


MichaelMcQuadeSeniorVicePresidentforScienceandTechnology

UnitedTechnologiesCorporation

BarbaraSchaalDeanoftheFacultyofArtsandSciencesMary-DellChiltonDistinguishedProfessorofBiology

WashingtonUniversityofSt.Louis

ChadMirkinGeorgeB.RathmannProfessorofChemistry

Director,InternationalInstituteforNanotechnology

NorthwesternUniversity

EricSchmidtExecutiveChairmanAlphabet,Inc.

MarioMolinaDistinguishedProfessor,ChemistryandBiochemistry

UniversityofCalifornia,SanDiegoProfessor,CenterforAtmosphericSciencesScrippsInstitutionofOceanography

DanielSchragSturgisHooperProfessorofGeologyProfessor,EnvironmentalScienceandEngineering

Director,HarvardUniversityCenterforEnvironment

HarvardUniversity

CraigMundiePresidentMundieAssociates

Staff

AshleyPredithExecutiveDirectorJenniferL.MichaelProgramSupportSpecialist

DianaE.PankevichAAASScience&TechnologyPolicyFellow


vii

PCASTForensicScienceinCriminalCourts:EnsuringScientificValidityofFeature-ComparisonMethods

WorkingGroup&SeniorAdvisorsWorkingGroupmembersparticipatedinthepreparationofaninitialdraftofthisreport.ThoseworkinggroupmemberswhoarenotPCASTmembersarenotresponsiblefor,nornecessarilyendorse,thefinalversionofthisreportasmodifiedandapprovedbyPCAST.

WorkingGroup

EricS.LanderPresidentBroadInstituteofHarvardandMIT

MichaelMcQuadeSeniorVicePresidentforScienceandTechnology

UnitedTechnologiesCorporation

S.JamesGates,Jr.JohnS.TollProfessorofPhysicsDirector,CenterforStringandParticleTheory

UniversityofMaryland,CollegePark

DanielSchragSturgisHooperProfessorofGeologyProfessor,EnvironmentalScienceandEngineering

Director,HarvardUniversityCenterforEnvironment

HarvardUniversity

SusanL.GrahamPehongChenDistinguishedProfessorEmeritainElectricalEngineeringandComputerScience


WilliamPressRaymerProfessorinComputerScienceandIntegrativeBiology

UniversityofTexasatAustin


viii

SeniorAdvisorCo-Chairs

TheHonorableHarryT.EdwardsJudgeUnitedStatesCourtofAppealsDistrictofColumbiaCircuit

JenniferL.MnookinDean,DavidG.PriceandDallasP.PriceProfessorofLaw,andFacultyCo-DirectorUniversityofCaliforniaLosAngelesLaw

SeniorAdvisors

TheHonorableJamesE.BoasbergDistrictJudgeUnitedStatesDistrictCourtDistrictofColumbia

TheHonorablePamelaHarrisJudgeUnitedStatesCourtofAppealsFourthCircuit

TheHonorableAndreM.DavisSeniorJudgeUnitedStatesCourtofAppealsFourthDistrict

KarenKafadarCommonwealthProfessorandChairDepartmentofStatisticsUniversityofVirginia

DavidL.FaigmanActingChancellor&DeanUniversityofCaliforniaHastingsCollegeoftheLaw

TheHonorableAlexKozinskiJudgeUnitedStatesCourtofAppealsNinthCircuit

StephenFienbergProfessorofStatisticsandSocialScienceCarnegieMellonUniversity

TheHonorableCorneliaT.L.PillardJudgeUnitedStatesCourtofAppealsDistrictofColumbiaCircuit

TheHonorableCharlesFriedBeneficialProfessorofLawHarvardLawSchoolHarvardUniversity

TheHonorableJedS.RakoffDistrictJudgeUnitedStatesDistrictCourtSouthernDistrictofNewYork

TheHonorableNancyGertnerSeniorLectureronLawHarvardLawSchoolHarvardUniversity

TheHonorablePattiB.SarisChiefJudgeUnitedStatesDistrictCourtDistrictofMassachusetts


ix

Staff

DianaE.PankevichAAASScience&TechnologyPolicyFellow

KristenZarrelliAdvisor,PublicPolicy&SpecialProjectsBroadInstituteofHarvardandMIT

Writer

TaniaSimoncelliSeniorAdvisortotheDirectorBroadInstituteofHarvardandMIT

EXECUTIVEOFFICEOFTHEPRESIDENT


x

PRESIDENT’SCOUNCILOFADVISORSONSCIENCEANDTECHNOLOGYWASHINGTON,D.C.20502

PresidentBarackObamaTheWhiteHouseWashington,DC20502DearMr.President:LETTERINSERTEDHERESincerely,

JohnP.Holdren EricS.LanderCo-Chair Co-Chair


xi


xii

TableofContentsThePresident’sCouncilofAdvisorsonScienceandTechnology.............................................v PCASTForensicScienceinCriminalCourts:EnsuringScientificValidityofFeature-Comparison

MethodsWorkingGroup&SeniorAdvisors.................................................................vii TableofContents.................................................................................................................xii ExecutiveSummary................................................................................................................1 1. Introduction.......................................................................................................................2 2. PreviousWorkonValidityofForensic-ScienceMethods.....................................................6

2.1 DNAEvidenceandWrongfulConvictions..................................................................................6 2.2 StudiesofSpecificForensic-ScienceMethodsandLaboratoryPractices..................................8 2.3 TestimonyConcerningForensicEvidence...............................................................................10 2.4 CognitiveBias..........................................................................................................................12 2.5 StateofForensicScience.........................................................................................................13 2.6 StateofForensicPractice........................................................................................................14 2.7 NationalResearchCouncilReport...........................................................................................15 2.8 RecentProgress.......................................................................................................................16

3. TheRoleofScientificValidityintheCourts......................................................................21 3.1 EvolutionofAdmissibilityStandards.......................................................................................21 3.2 FoundationalValidityandValidityasApplied.........................................................................23

4. ScientificCriteriaforValidityandReliabilityofForensicFeature-ComparisonMethods....25 4.1 Feature-ComparisonMethods:ObjectiveandSubjectiveMethods........................................27 4.2FoundationalValidity:RequirementforEmpiricalStudies......................................................28 4.3FoundationalValidity:RequirementforScientificallyValidTestimony...................................35 4.4NeitherExperiencenorProfessionalPracticesCanSubstituteforFoundationalValidity.......36 4.5ValidityasApplied:KeyElements............................................................................................36 4.6ValidityasApplied:ProficiencyTesting....................................................................................37 4.7Non-EmpiricalViewsintheForensicCommunity....................................................................40 4.8EmpiricalViewsintheForensicCommunity............................................................................43 4.9SummaryofScientificFindings................................................................................................46

5. EvaluationofScientificValidityforSevenFeature-ComparisonMethods.........................48 5.1DNAanalysisofsingle-sourceandsimple-mixturesamples....................................................50 5.2DNAanalysisofcomplex-mixturesamples..............................................................................56 5.3BitemarkAnalysis.....................................................................................................................61 5.4LatentFingerprintAnalysis.......................................................................................................66 Table1:Errorratesinstudiesoflatentprintanalysis......................................................................76 5.5FirearmsAnalysis.....................................................................................................................82 5.6FootwearAnalysis:IdentifyingCharacteristics........................................................................90 5.7HairAnalysis.............................................................................................................................94 5.8Applicationtoadditionalmethods...........................................................................................97 5.9Conclusion................................................................................................................................97


xiii

6. ActionstoEnsureScientificValidityinForensicScience:RecommendationstoScience-basedAgencies.............................................................................................................99 6.1RoleforNISTinongoingevaluationoffoundationalvalidity...................................................99 6.2Acceleratingthedevelopmentofobjectivemethods............................................................100 6.3ImprovingOSAC.....................................................................................................................101 6.4NeedforanR&DstrategyforForensicScience.....................................................................102 6.5Recommendations.................................................................................................................103

7. ActionstoEnsureScientificValidityinForensicScience:RecommendationstotheFBILaboratory..................................................................................................................106 7.1RoleforFBILaboratory..........................................................................................................106 7.2Recommendations.................................................................................................................109

8. ActionstoEnsureScientificValidityinForensicScience:RecommendationstotheAttorneyGeneral.......................................................................................................................111 8.1Ensuringtheuseofscientificallyvalidmethodsinprosecutions...........................................111 8.2RevisionofDOJRecentlyProposedGuidelinesonExpertTestimony....................................111 8.3Recommendations.................................................................................................................115

9. ActionstoEnsureScientificValidityinForensicScience:RecommendationstotheJudiciary...................................................................................................................................117 9.1Scientificvalidityasafoundationforadmissibilityofexperttestimony...............................117 9.2Roleofpastprecedent...........................................................................................................118 9.3Resourcesforjudges..............................................................................................................119 9.4Recommendations.................................................................................................................120

10. ScientificFindings.......................................................................................................121 AppendixA:StatisticalIssues..............................................................................................126

Sensitivityandfalsepositiverate...................................................................................................126 ConfidenceIntervals.......................................................................................................................127 CalculatingResultsforConclusiveTests.........................................................................................128 BayesianAnalysis............................................................................................................................128

AppendixB.AdditionalExpertsProvidingInput..................................................................130


1

ExecutiveSummaryTOBEINSERTEDHERE


2

1. Introduction

“Forensicscience”hasbeendefinedastheapplicationofscientificortechnicalpracticestotherecognition,collection,analysis,andinterpretationofevidenceforcriminalandcivillaworregulatoryissues.1Theforensicsciencesencompassabroadrangeofdisciplines,eachwithitsownsetoftechnologiesandpractices.TheNationalInstituteofJustice(NIJ)dividesthosedisciplinesintotwelvecategories:generaltoxicology;firearmsandtoolmarks;questioneddocuments;traceevidence(suchashairandfiberanalysis);controlledsubstances;biological/serologyscreening(includingDNAanalysis);firedebris/arsonanalysis;impressionevidence;bloodpatternevidence;crimesceneinvestigation;medicolegaldeathinvestigation;anddigitalevidence.2Intheyearsahead,scienceandtechnologywilllikelyofferadditionalpowerfultoolsfortheforensicdomain—perhapstheabilitytocomparepopulationsofbacteriainthegutorpatternsofsearchontheinternet.

Historically,forensicsciencehasbeenusedprimarilyintwophasesofthecriminal-justiceprocess:(1)investigation,whichseekstoidentifythelikelyperpetratorofacrime,and(2)prosecution,whichseekstoprovetheguiltofadefendantbeyondareasonabledoubt.(Inrecentyears,forensicscience—particularlyDNAanalysis—hasalsocomeintowideuseforchallengingpastconvictions.)Importantly,theinvestigativeandprosecutorialphasesinvolvedifferentstandardsfortheuseofforensicscienceandotherinvestigativetools.Ininvestigations,insightsandinformationmaycomefrombothwell-establishedscienceandexploratoryapproaches.3Intheprosecutionphase,forensicsciencemustsatisfyahigherstandard.Specifically,theFederalRulesofEvidencerequirethatexperttestimonybebased,amongotherthings,on“reliableprinciplesandmethods”thathavebeen“reliablyapplied”tothefactsofthecase.4And,theSupremeCourthasheldthatjudgesmustdetermine“whetherthereasoningormethodologyunderlyingthetestimonyisscientificallyvalid.”5

Thisiswherelegalstandardsandscientificstandardsintersect.Judges’decisionsabouttheadmissibilityofscientificevidencerestsolelyonlegalstandards;theyareexclusivelytheprovinceofthecourts.But,theoverarchingsubjectofthejudges’inquiryisscientificvalidity.6Itistheproperprovincethescientificcommunitytoprovideguidanceconcerningscientificstandardsforscientificvalidity.7

1Definitionof“forensicscience”asprovidedbytheNationalCommissiononForensicScienceinitsViewsDocument,“Definingforensicscienceandrelatedterms.”AdoptedApril30-May1,2015.www.justice.gov/ncfs/file/786571/download.2See:NationalInstituteofJustice.StatusandNeedsofForensicScienceServiceProviders:AReporttoCongress.2006.www.ojp.usdoj.gov/nij/pubs-sum/213420.htm.3WhileinvestigativemethodsneednotmeetthestandardsofreliabilityrequiredundertheFederalRulesofEvidence,theyshouldbebasedinsoundscientificprinciplesandpracticessoastoavoidfalseaccusations.4Fed.R.Evid.702.5Daubertv.MerrellDowPharmaceuticals509U.S.579(1993)at592.6Daubert,at594.7Inthisreport,PCASTaddressessolelythescientificstandardsforscientificvalidityandreliability.Wedonotofferopinionsconcerninglegalstandards.


3

Afocusonthescientificsideofthisintersectionistimelybecauseithasbecomeincreasinglyclearinrecentyearsthatlackofrigorintheassessmentofthescientificvalidityofforensicevidenceisnotjustahypotheticalproblembutarealandsignificantweaknessinthejudicialsystem.AsrecountedinChapter2,reviewsbycompetentbodiesofthescientificunderpinningsofforensicdisciplinesandtheuseincourtroomsofevidencebasedonthosedisciplineshaverevealedadismayingfrequencyofinstancesofuseofforensicevidencethatdonotpassanobjectivetestofscientificvalidity.

ThemostcomprehensivesuchreviewtodatewasconductedbyaNationalResearchCouncil(NRC)committeeco-chairedbyJudgeHarryEdwardsoftheU.S.CourtofAppealsfortheDistrictofColumbiaCircuitandConstantineGatsonis,DirectoroftheCenterforStatisticalSciencesatBrownUniversity.MandatedbyCongressinanappropriationsbillsignedintolawinlate2005,thestudygotunderwayinthefallof2006andthecommitteereleaseditsreportinFebruary2009.8

Thereportdescribedadisturbingpatternofdeficienciescommontomanyoftheforensicmethodsroutinelyusedinthecriminaljusticesystem,mostimportantlyalackofrigorousandappropriatestudiesestablishingtheirscientificvalidity,concludingthat“muchforensicevidence—including,forexample,bitemarksandfirearmandtoolmarkidentifications—isintroducedincriminaltrialswithoutanymeaningfulscientificvalidation,determinationoferrorrates,orreliabilitytestingtoexplainthelimitsofthediscipline.”9

In2013,afterprolongeddiscussionoftheNRCreport’sfindingsandrecommendationsinsideandoutsidetheFederalgovernment,theDepartmentofJustice(DOJ)—incollaborationwiththeNationalInstituteofStandardsandTechnology(NIST)—establishedtheNationalCommissiononForensicScience(NCFS)asaFederaladvisorybodychargedwithprovidingforensic-scienceguidanceandpolicyrecommendationstotheAttorneyGeneral.Co-chairedbytheDeputyAttorneyGeneralandtheDirectorofNIST,theNCFS’s32membersincludeeightacademicscientistsandfiveothersciencePhDs;theothermembersincludejudges,attorneysandforensicpractitioners.Tostrengthenforensicsciencemoregenerally,in2014,NISTestablishedtheOrganizationforScientificAreaCommitteesforForensicScience(OSAC)to“coordinatedevelopmentofstandardsandguidelines…toimprovequalityandconsistencyofworkintheforensicsciencecommunity.”

InSeptember2015,PresidentObamaaskedhisCouncilofAdvisorsonScienceandTechnology(PCAST)toexplore,inlightoftheworkbeingdonebytheNCSFandOSAC,whatadditionaleffortscouldcontributetostrengtheningtheforensic-sciencedisciplinesandensuringthescientificreliabilityofforensicevidenceusedintheNation’slegalsystem.Afterreviewoftheongoingactivitiesandtherelevantscientificandlegalliteratures—includingparticularlythescientificandlegalassessmentsinthe2009NRCreport—PCASTconcludedthatthemostusefulcontributionitcouldmakewouldbetoaddclarityonthescientificmeaningof“reliableprinciplesandmethods”and“scientificvalidity”inthecontextofcertainforensicdisciplines.

8NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.xx.9Ibid,pp.107-8.


4

Withinthebroadspanofforensicdisciplines,wechosetonarrowourfocustotechniquesthatwerefertohereasforensic“feature-comparison”methods(seeBox1).10Whileonemotivationforthisnarrowingwastomakeourtasktractablewithinthelimitsofavailabletimeandresources,wechosethisparticularclassofmethodsbecause:(1)theyarecommonlyusedincriminalcases;(2)theyhaveattractedahighdegreeofconcernwithrespecttovalidity(see,e.g.,the2009NRCreport);and(3)theyallbelongtothesamebroadscientificdiscipline,metrology,whichis“thescienceofmeasurementanditsapplication”,inthiscasetomeasuringandcomparingfeatures.11

BOX1.Forensicfeature-comparisonmethods

PCASTusestheterm“forensicfeature-comparisonmethods”torefertothewidevarietyofmethodsthataimtodeterminewhetheranevidentiarysample(e.g.,fromacrimescene)isorisnotassociatedwithapotentialsourcesample(e.g.,fromasuspect)basedonthepresenceofsimilarpatterns,impressions,features,orcharacteristicsinthesampleandthesource.Examplesincludetheanalysisof:DNA,hair,latentfingerprints,firearmsandspentammunition,toolandtoolmarks,shoeprintsandtiretracks,bitemarks,andhandwriting.

PCASTbeganthisstudybyformingaworkinggroupofsixofitsmemberstogatherinformationforconsideration.12Toeducateitselfaboutfactualmattersrelatingtotheinteractionbetweenscienceandlaw,PCASTconsultedwithapanelofSeniorAdvisors(listedinthefrontmatter)comprisingninecurrentorformerFederaljudges,oneformerU.S.SolicitorGeneralandStatesupremecourtjustice,twolawschooldeans,andtwostatisticians,whohaveexpertiseinthisdomain.PCASTalsosoughtinputfromadiversegroupofadditionalexpertsandstakeholders,includingforensicscientistsandpractitioners,judges,prosecutors,defenseattorneys,criminaljusticereformadvocates,statisticians,academicresearchers,andFederalagencyrepresentatives(seeAppendixB).Inputwasgatheredthroughmultiplein-personmeetingsandconferencecalls,includingasessionatameetingofPCASTonJanuary15,2016.PCASTalsotooktheunusualstepofinitiatinganonline,opensolicitationtobroadeninput,inparticularfromtheforensic-sciencepractitionercommunity;morethan70responseswerereceived.13

10PCASTnotesthatthereareissuesrelatedtothescientificvalidityofothertypesofevidence—includingnotablyarsonscienceandabusiveheadtraumacommonlyreferredtoas“ShakenBabySyndrome”—that,althoughbeyondthescopeofthisreport,requireurgentattention.11InternationalVocabularyofMetrology–BasicandGeneralConceptsandAssociatedTerms(VIM3rdedition)JCGM200:2012.12Twoofthemembershavebeeninvolvedwithforensicscience.PCASTCo-chairEricLanderhasservedinvariousscientificroles(expertwitnessinPeoplev.Castro545N.Y.S.2d985(Sup.Ct.1989),aseminalcaseonthequalityofDNAanalysisdiscussedonp.34;court’switnessinU.S.v.Yee,134F.R.D.161in1991;memberoftheNRCpanelonforensicDNAanalysisin1992;scientificco-authorwithaforensicscientistfromtheFBILaboratoryin1994;andamemberoftheBoardofDirectorsoftheInnocenceProjectfrom2004tothepresent).Alloftheseroleshavebeenunremunerated.PCASTmemberS.JamesGates,Jr.hasbeenamember,sinceitsinception,oftheNationalCommissiononForensicScience.13See:www.whitehouse.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensics_request_for_information.pdf.


5

PCASTalsosharedadraftofthisreportwithNISTandDOJ,whichprovideddetailedandhelpfulcommentsthatwerecarefullyconsideredinrevisingthereport.

PCASTexpressesitsgratitudetoallthosewhosharedtheirviews.TheirwillingnesstoengagewithPCASTdoesnotimplyendorsementoftheviewsexpressedinthereport.Responsibilityfortheopinions,findingsandrecommendationsexpressedinthereportandforanyerrorsoffactorinterpretationrestssolelywithPCAST.

Theremainderofourreportisorganizedasfollows.

• Chapter2providesabriefoverviewofthefindingsofotherstudiesrelatingtoforensicpracticeandtestimonybasedonit,anditreviews,aswell,Federalactionscurrentlyunderwaytostrengthenforensicscience.

• Chapter3brieflyreviewstheroleofscientificvaliditywithinthelegalsystem.Itdescribestheimportantdistinctionbetweenlegalstandardsandscientificstandards.

• Chapter4thendescribesthescientificstandardsfor“reliableprinciplesandmethods”and“scientificvalidity,”astheyapplytoforensicfeature-comparisonmethods,andoffersclearcriteriathatcouldbereadilyappliedbycourts.

• Chapter5illustratestheapplicationoftheindicatedcriteriabyusingthemtoevaluatethescientificvalidityofsiximportant“feature-comparison”methods:DNAanalysisofsingle-sourceandsimple-mixturesamples,DNAanalysisofcomplexmixtures,bitemarkanalysis,latentfingerprintanalysis,firearmsanalysis,andfootwearanalysis.Wealsodiscussanevaluationbyothersofaseventhmethod,hairanalysis.

• InChapters6-9,weofferrecommendations,basedonthefindingsofChapters4-5,concerningFederalactionsthatcouldbetakentostrengthenforensicscienceandpromoteitsmorerigoroususeinthecourtroom.


6

2. PreviousWorkonValidityofForensic-ScienceMethods

Developmentsoverthepasttwodecades—includingtheexonerationofdefendantswhohadbeenwrongfullyconvictedbasedinpartonforensic-scienceevidence,avarietyofstudiesofthescientificunderpinningsoftheforensicdisciplines,reviewsofexperttestimonybasedonforensicfindings,andscandalsinstatecrimelaboratories—havecalledincreasingattentiontothequestionofthevalidityandreliabilityofsomeimportantforensicmethodsevidenceandtestimonybaseduponthem.(Fordefinitionsofkeytermssuchasvalidityandreliability,seeBox1onpage4.)

Inthischapter,webrieflyreviewthishistorytosetthestageforourownassessmentofthecurrentpictureandthepathforward.14

2.1 DNAEvidenceandWrongfulConvictionsIronically,itwastheemergenceandmaturationofanewforensicscience,DNAanalysis,thatfirstledtoseriousquestioningofthevalidityofmanyofthetraditionalforensicdisciplines.ThathappenedwhendefendantsconvictedwiththehelpofforensicevidencefromthosetraditionaldisciplinesbegantobeexoneratedonthebasisofpersuasiveDNAcomparisons.Howthiscametopassprovidesusefulcontextforourinquiryhere.

WhenDNAevidencewasfirstintroducedinthecourts,beginninginthelate1980s,itwasinitiallyhailedasinfallible.Butthemethodsusedinearlycasesturnedouttobeunreliable:testinglabslackedvalidatedandconsistently-appliedproceduresfordefiningDNApatternsfromsamples,fordeclaringwhethertwopatternsmatchedwithinagiventolerance,andfordeterminingtheprobabilityofsuchmatchesarisingbychanceinthepopulation.15

WhenDNAevidencewasdeclaredinadmissibleinPeoplev.Castro,aNewYorkcasein1989,scientists—includingattheU.S.NationalAcademyofSciencesandtheFederalBureauofInvestigation(FBI)—cametogether

14Inproducingthissummarywehavereliedparticularlyonthe2009reportonforensicsciencefromtheNationalResearchCouncil(www.nap.edu/download/12589)andtheNationalAcademiesofSciences,Engineering,andMedicine2015NRCreport,SupportforForensicScienceResearch:ImprovingtheScientificRoleoftheNationalInstituteofJustice.TheNationalAcademiesPress.WashingtonDC.(2015)(www.nap.edu/download/21772).15Lander,E.S.“DNAfingerprintingontrial.”Nature,Vol.339(1989):501-5.;Lander,E.S.andBudowle,B.“DNAfingerprintingdisputelaidtorest.”Nature,Vol.371(1994):735-8.;Kaye,D.H.“DNAEvidence:Probability,PopulationGenetics,andtheCourts.”Harv.J.L.&Tech,Vol7(1993);101-72.;Roberts,L.“FighteruptsoverDNAfingerprinting.”Science,Vol.254(1991):1721-3.;Thompson,W.C.andFord,S.“IsDNAfingerprintingreadyforthecourts?”NewScientist,Vol.125(1990):p.38-43.;Neufeld,P.J.,Colman,N.“Whensciencetakesthewitnessstand.”ScientificAmerican,Vol.262(1991):p.46-53.


7

topromotethedevelopmentofreliableprinciplesandmethodsthathaveenabledDNAanalysisofsingle-sourcesamplestobecomethe“goldstandard”offorensicscienceforbothinvestigationandprosecution.16

BoththeinitialrecognitionofseriousproblemsandthesubsequentdevelopmentofreliableprocedureswereaidedbytheexistenceofarobustcommunityofmolecularbiologistswhousedDNAanalysisinnon-forensicapplications,suchasinbiomedicalandagriculturalsciences.Theywerealsoaidedbyjudgeswhorecognizedthatthispowerfulforensicmethodshouldonlybeadmittedascourtroomevidenceonceitsreliabilitywasproperlyestablished.

OnceDNAanalysisbecameareliablemethodology,thepowerofthetechnology—includingitsabilitytoanalyzesmallsamplesandtodistinguishbetweenindividuals—madeitpossiblenotonlytoidentifyandconvicttrueperpetrators,butalsotoclearmistakenlyaccusedsuspectsbeforeprosecutionandtore-examineanumberofpastconvictions.ReviewsbytheNIJ17andothershavefoundthatDNAtestingduringthecourseofinvestigationshasclearedtensofthousandsofsuspects.DNA-basedre-examinationofpastcases,moreover,hasledsofartotheexonerationsof342defendants,including20whohadbeensentencedtodeath,andtotheidentificationof147realperpetrators.18

Independentreviewsofthesecaseshaverevealedthatmanyreliedinpartonfaultyexperttestimonyfromforensicscientistswhohadtoldjuriesthatsimilarfeaturesinapairofsamplestakenfromasuspectandfromacrimescene(hair,bullets,bitemarks,tireorshoetreads,orotheritems)implicateddefendantsinacrimewithahighdegreeofcertainty.19Accordingtothereviews,theseerrorswerenotsimplyamatterofindividualexaminerstestifyingtoconclusionsthatturnedouttobeincorrect;rather,theyreflectedasystemicproblem—

16Peoplev.Castro545N.Y.S.2d985(Sup.Ct.1989).Thecase,inwhichajanitorwaschargedwiththemurderofawomanintheBronx,wasamongthefirstcriminalcasesinvolvingDNAanalysisintheUnitedStates.Thecourthelda15-week-longpretrialhearingabouttheadmissibilityoftheDNAevidence.Bytheendofthehearing,theindependentexpertsforboththedefenseandprosecutionunanimouslyagreedthattheDNAevidencepresentedwasnotscientificallyreliable—andthejudgeruledtheevidenceinadmissible.See:Lander,E.S.“DNAfingerprintingontrial.”Nature,Vol.339(1989):501-5.TheseeventseventuallyledtotwoNRCreportsonforensicDNAanalysis,in1992and1996,andtothefoundingoftheInnocenceProject(www.innocenceproject.org).17DNAtestinghasexcluded20-25%ofinitialsuspectsinsexualassaultcases.U.SDepartmentofJustice,OfficeofJusticePrograms,NationalInstituteofJustice.ConvictedbyJuries,ExoneratedbyScience:CaseStudiesintheUseofDNAEvidencetoEstablishInnocenceafterTrial,(1996):p.xxviii.18InnocenceProject,“DNAExonerationsintheUnitedStates.”See:www.innocenceproject.org/dna-exonerations-in-the-united-states.19Forexample,see:Gross,S.R.,andM.Shaffer.“ExonerationsintheUnitedStates,1989-2012.”NationalRegistryofExonerations,(2012)availableat:www.law.umich.edu/special/exoneration/Documents/exonerations_us_1989_2012_full_report.pdf.Seealso:MichaelSaks,M.J.,andJ.J.Koehler.,“Thecomingparadigmshiftinforensicidentificationscience.“ForensicIdentificationScience,Vol.,”309,No.5736Science892(2005):892-5.).


8

thetestimonywasbasedonmethodsandincludedclaimsofaccuracythatwerecloakedinpurportedscientificrespectabilitybutactuallyhadneverbeensubjectedtomeaningfulscientificscrutiny.20

2.2 StudiesofSpecificForensic-ScienceMethodsandLaboratoryPracticesThequestionsthatDNAanalysishadraisedaboutthescientificvalidityoftraditionalforensicdisciplinesandtestimonybasedonthemled,naturally,toincreasedeffortstotestempiricallythereliabilityofthemethodsthatthosedisciplinesemployed.Scrutinywasdirected,similarly,tothepracticesbywhichforensicevidenceiscollected,stored,andanalyzedincrimelaboratoriesaroundthecountry.TheFBICrimeLaboratory,widelyregardedasoneofthebestinthecountry,playedanimportantroleinthelatterinvestigations,re-assessingitsownpracticesaswellasthoseofothers.Inwhatfollowswesummarizesomeofthekeyfindingsofthestudiesofmethodsandpracticesthatensuedinthecaseofthe“comparison”disciplinesthatareourfocusinthisreport.

BulletLeadExamination

Fromthe1960suntil2005,theFBIusedcompositionalanalysisofbulletleadasaforensictoolofanalysistoidentifythesourceofbullets.Yet,anNRCreportcommissionedbytheFBIandreleasedin2004challengedthefoundationalvalidityofidentificationsbasedonthediscipline.Thetechniqueinvolvedcomparingthequantityofvariouselementsinbulletsfoundatacrimescenewiththatofunusedbulletstodeterminewhetherthebulletscamefromthesameboxofammunition.The2004NRCreportfoundthatthereisnoscientificbasisformakingsuchadetermination.21Whilethemethodfordeterminingtheconcentrationsofdifferentelementswithinabulletwasfoundtobereliable,thereportfoundtherewasinsufficientresearchanddatatosupportdrawingaconnection,basedoncompositionalsimilaritybetweenaparticularbulletandagivenbatchofammunition,whichisusuallytherelevantquestioninacriminalcase.22In2005,theFBIannouncedthatitwoulddiscontinuethepracticeofbulletleadexaminations,notingthatwhileit“firmlysupportsthescientific

20Garrett,B.L.,andP.J.Neufeld.“Invalidforensicsciencetestimonyandwrongfulconvictions.”VirginiaLawReview,Vol.91,No.1(2009):1-97;NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):pp.42-3.21Leadbulletexamination,alsoknownasCompositionalAnalysisofBulletLead(CABL),involvescomparingtheelementalcompositionofbulletsfoundatacrimescenewithunusedcartridgesinthepossessionofasuspect.Thistechniqueassumesthat(1)themoltensourceusedtoproduceasingle“lot”ofbulletshasauniformcompositionthroughout;(2)notwomoltensourceshavethesamecomposition;and(3)bulletswithdifferentcompositionsarenotmixedduringthemanufacturingorshippingprocesses.However,inpractice,thisisnotthecase.The2004NRCreportfoundthatcompositionallyindistinguishablevolumesofleadcouldproducesmalllotsofbullets—ontheorderof12,000bullets—orlargelots—withmorethan35millionbullets.Thereportalsofoundnoassurancethatindistinguishablevolumesofleadcouldnotoccuratdifferenttimesandplaces.Neitherscientistsnorbulletmanufacturersareabletodefinitivelyattesttothesignificanceofanassociationmadebetweenbulletsinthecourseofabulletleadexamination.ThemostthatonecansayisthatbulletsthatareindistinguishablebyCABLcouldhavecomefromthesamesource.NationalResearchCouncil.ForensicAnalysis:WeighingBulletLeadEvidence.TheNationalAcademiesPress.WashingtonDC.2004.22Faigman,D.L.,Kaye,D.H.,Saks,M.J.,andJ.Sanders(Eds).ModernScientificEvidence:TheLawandScienceofExpertTestimony,2015-2016ed.Thomson/WestPublishing(2016).


9

foundationofbulletleadanalysis,”themanufacturinganddistributionofbulletswastoovariabletomakethematchingreliable.23

LatentFingerprints

In2005,aninternationalcommitteeestablishedbytheFBIreleasedareportconcerningflawsintheFBI’spracticesforfingerprintidentificationthathadledtoaprominentmisidentification.Basedalmostentirelyonalatentfingerprintrecoveredfromthe2004bombingoftheMadridcommutertrainsystem,theFBIerroneouslydetainedanAmericaninPortland,Oregonandheldhimfortwoweeksasamaterialwitness.24AnFBIexaminerconcludedthefingerprintsmatchedwith“100%certainty,”althoughSpanishauthoritieswereunabletoconfirmthematch.25ThereviewcommitteeconcludedthattheFBI’smisidentificationhadoccurredprimarilyasaresultof“confirmationbias.”26Similarly,areportbytheDOJ’sOfficeoftheInspectorGeneralhighlighted“reversereasoning”fromtheknownprinttothelatentimagethatledtoanexaggeratedfocusonapparentsimilaritiesandinadequateattentiontodifferencesbetweentheimages.27

HairAnalysis

In2002,FBIscientistsusedmitochondrialDNAsequencingtore-examine170microscopichaircomparisonsthattheagency’sscientistshadperformedincriminalcases.TheDNAanalysisshowedthat,in11%ofcasesinwhichtheFBIexaminershadfoundthehairsamplestomatchmicroscopically,DNAtestingofthesamplesrevealedtheyactuallycamefromdifferentindividuals.28Thesefalseassociationsmaynothavebeentheresultofafailureoftheexaminertoperformtheanalysiscorrectly;instead,thecharacteristicscouldhavejusthappenedtohavebeensharedbychance.Thestudyshowedthatthepowerofmicroscopichaircomparisontodistinguishbetweensamplesfromdifferentsourceswasmuchlowerthanpreviouslyassumed.(Forexample,earlierstudiessuggestedthatthefalsepositiverateforofhairanalysisisintherangeof1in40,000.29)

23FederalBureauofInvestigation.FBILaboratoryAnnouncesDiscontinuationofBulletLeadExaminations.(September1,2005,pressrelease).www.fbi.gov/news/pressrel/press-releases/fbi-laboratory-announces-discontinuation-of-bullet-lead-examinations(accessedMay6,2016).24Stacey,R.B.“Reportontheerroneous fingerprint individualization in theMadrid trainbombingcase.”ForensicScienceCommunications,Vol.7,No.1(2005).25ApplicationforMaterialWitnessOrderandWarrantRegardingWitness:BrandonBieriMayfield,InreFederalGrandJuryProceedings03-01,337F.Supp.2d1218(D.Or.2004)(No.04-MC-9071).26Specifically,similaritiesbetweenthetwoprints,combinedwiththeinherentpressureofworkingonanextremelyhigh-profilecase,influencedtheinitialexaminer’sjudgment:ambiguouscharacteristicswereinterpretedaspointsofsimilarityanddifferencesbetweenthetwoprintswereexplainedaway.Asecondexaminer,notshieldedfromthefirstexaminer’sconclusions,simplyconfirmedthefirstexaminer’sresults.See:Stacey,R.B.“ReportontheerroneousfingerprintindividualizationintheMadridtrainbombingcase.”ForensicScienceCommunications,Vol.7,No.1(2005).27U.S.DepartmentofJustice,OfficeoftheInspectorGeneral.“AreviewoftheFBI’shandlingoftheBrandonMayfieldcase.”(2006).oig.justice.special/s0601/final.pdf.28Houck,M.M.,andB.Budowle.“CorrelationofmicroscopicandmitochondrialDNAhaircomparisons.”JournalofForensicSciences,Vol.47,No.5(2002):964-7.29Gaudette,B.D.,Keeping,E.S.“Anattemptatdeterminingprobabilitiesinhumanscalphaircomparisons.“JournalofForensicSciences,Vol.19(1975):599-606.ThisstudywasrecentlycitedbyDOJtosupporttheassertionthathairanalysisis


10

Bitemarks

A2010studyofexperimentallycreatedbitemarksproducedbyknownbitersfoundthatskindeformationdistortsbitemarkssosubstantiallyandsovariablythatcurrentproceduresforcomparingbitemarksareunabletoreliablyexcludeorincludeasuspectasapotentialbiter.(“Thedataderivedshowednocorrelationandwasnotreproducible,thatis,thesamedentitioncouldnotcreateameasurableimpressionthatwasconsistentinalloftheparametersinanyofthetestcircumstances.”)30ArecentstudybytheAmericanBoardofForensicOdontologyalsoshowedadisturbinglackofconsistencyinthewaythatforensicodontologistsgoaboutanalyzingbitemarks,includingevenondecidingwhethertherewassufficientevidencetodeterminewhetheraphotographedbitemarkwasahumanbitemark.31InFebruary2016,followingasix-monthinvestigation,theTexasForensicScienceCommissionunanimouslyrecommendedamoratoriumontheuseofbitemarkidentificationsincriminaltrials,concludingthatthevalidityofthetechniquehasnotbeenscientificallyestablished.32

Theseexamplesillustratehowmanyforensicfeature-comparisonmethodsthathavebeeninwideusehavenonethelessnotbeensubjectedtomeaningfultestsofscientificvalidityormeasuresofreliability.

2.3 TestimonyConcerningForensicEvidence

Reviewsoftrialtranscriptshavefoundthatexpertwitnesseshaveoftenoverstatedtheprobativevalueoftheirevidence,goingfarbeyondwhattherelevantsciencecanjustify.Forexample,someexaminershavetestified:

• thattheirconclusionsare“100%certain;”have“zero,”“essentiallyzero,”vanishinglysmall,”“negligible,”“minimal”or“microscopic”errorrate;orhaveachanceoferrorsoremoteastobea“practicalimpossibility.”33Asmanyreviewshavenoted,however,suchstatementsarenot

avalidandreliablescientificmethodology.www.justice.gov/dag/file/877741/download.ThetopicofhairanalysisisdiscussedinChapter5.30Bush,M.A.,Cooper,H.I.,andR.B.Dorion.“Inquiryintothescientificbasisforbitemarkprofilingandarbitrarydistortioncompensation.”JournalofForensicSciences,Vol.55,No.4(2010):976-83.SeealsoBush,M.A.,Miller,R.G.,Bush,P.J.,andR.B.Dorion.“Biomechanicalfactorsinhumandermalbitemarksinacadavermodel.”JournalofForensicSciences,Vol.54,No.1(2009):167-76.31Balko,R.“Abitemarkmatchingadvocacygroupjustconductedastudythatdiscreditsbitemarkevidence.”WashingtonPost,April8,2015.www.washingtonpost.com/news/the-watch/wp/2015/04/08/a-bite-mark-matching-advocacy-group-just-conducted-a-study-that-discredits-bite-mark-evidence.;AdamJ.Freeman&IainA.Pretty,ConstructValidityofBitemarkAssessmentsUsingtheABOBitemarkDecisionTree,AmericanAcademyofForensicSciences,AnnualMeeting,OdontologySection,G14,February2015(datamadeavailablebytheauthorsuponrequest).ThedatawereanalyzedbyamemberofthePanelofSeniorAdvisorsandreviewedbyPCAST.32TexasForensicScienceCommission.“ForensicbitemarkcomparisoncomplaintfiledbyNationalInnocenceProjectonbehalfofStevenMarkChaney–FinalReport.”(2016).www.fsc.texas.gov/sites/default/files/FinalBiteMarkReport.pdf.33Thompson,W.C.,F.Taroni,andC.G.G.Aitken.“HowtheProbabilityofaFalsePositiveAffectstheValueofDNAEvidence.”JForensicSci, Vol.48,No.1(2003):1-8.;Thompson,W.C.“TheMythofInfallibility”,InSheldonKrimsky&JeremyGruber(Eds.)GeneticExplanations:SenseandNonsense,HarvardUniversityPress(2013);Cole,S.A.“Morethanzero:Accountingforerrorinlatentfingerprintidentification.”JournalofCriminalLawandCriminology,Vol.95,No.3(2005):

Dr4ensic

Highlight

Dr4ensic

Highlight

Dr4ensic

Highlight

Dr4ensic

Highlight

Dr4ensic

Highlight

Dr4ensic

Highlight


11

scientificallydefensible.Alllaboratorytestsandfeature-comparisonanalyseshavenon-zeroerrorrates,evenifanexaminerreceivedaperfectscoreonaparticularperformancetestinvolvingalimitednumberofsamples.34Evenhighlyautomatedtestsdonothaveazeroerrorrate.35,36

• thattheycan“individualize”evidence—forexample,usingmarkingsonabullettoattributeittoaspecificweapon“totheexclusionofeveryotherfirearmintheworld”—anassertionthatisnotsupportablebytherelevantscience.37

• thataresultistrue“toareasonabledegreeofscientificcertainty.”Thisphrasehasnogenerallyacceptedmeaninginscienceandisopentowidelydifferinginterpretationsbydifferentscientists.38Moreover,thestatementmaybetakenasimplying“certainty.”

DOJReviewofTestimonyonHairAnalysis

In2012,theDOJandFBIannouncedthattheywouldinitiateaformalreviewoftestimonyinmorethan3,000criminalcasesinvolvingmicroscopichairanalysis.Initialresultsofthisunprecedentedreview,conductedinconsultationwiththeInnocenceProjectandtheNationalAssociationofCriminalDefenseLawyers,foundthatFBIexaminershadprovidedscientificallyinvalidtestimonyinmorethan95percentofcaseswhereexaminer-providedtestimonywasusedtoinculpateadefendantattrial.Theseproblemsweresystemic:26ofthe28FBIhairexaminerswhotestifiedinthe328casesprovidedscientificallyinvalidtestimony.39,40

985-1078;andKoehler,J.J.“Forensicsorfauxrensics?Testingforaccuracyintheforensicsciences.”papers.ssrn.com/sol3/papers.cfm?abstract_id=2773255(accessedJune28,2016).34Cole,S.A.“Morethanzero:Accountingforerrorinlatentfingerprintidentification.”JournalofCriminalLawandCriminology,Vol.95,No.3(2005):985-1078andKoehler,J.J.“Forensicsorfauxrensics?Testingforaccuracyintheforensicsciences.”papers.ssrn.com/sol3/papers.cfm?abstract_id=2773255(accessedJune28,2016).35Thompson,W.C.,Franco,T,andC.G.G.Aitken.“HowtheprobabilityofafalsepositiveaffectsthevalueofDNAevidence.”JournalofForensicScience,Vol.48,No.1(2003):1-8.36AswediscussinChapter4,falsepositiveresultscanarisefromtwosources:(1)similaritybetweentwofeaturesthatoccurbychanceand(2)human/technicalfailures.37NationalResearchCouncil.BallisticImaging.TheNationalAcademiesPress.WashingtonDC.2008.Seealso:Saks,M.J.andJ.J.Koehler.“Theindividualizationfallacyinforensicscienceevidence.”ForensicScienceEvidence.”VanderbiltLawReview,Vol.61,No.1(2008):199-218.38NationalCommissiononForensicScience,“RecommendationstotheAttorneyGeneralRegardingUseoftheTerm‘ReasonableScientificCertainty’,”ApprovedMarch22,2016,availableat:www.justice.gov/ncfs/file/839726/download.TheNCSFstatesthat“forensicdisciplineconclusionsareoftentestifiedtoasbeingheld‘toareasonabledegreeofscientificcertainty’or‘toareasonabledegreeof[discipline]certainty.’Thesetermshavenoscientificmeaningandmaymisleadfactfindersaboutthelevelofobjectivityinvolvedintheanalysis,itsscientificreliabilityandlimitations,andtheabilityoftheanalysistoreachaconclusion.”39FederalBureauofInvestigation.FBITestimonyonMicroscopicHairAnalysisContainedErrorsinatLeast90PercentofCasesinOngoingReview,(April20,2015,pressrelease).www.fbi.gov/news/pressrel/press-releases/fbi-testimony-on-microscopic-hair-analysis-contained-errors-in-at-least-90-percent-of-cases-in-ongoing-review.40Theerroneousstatementsfellintothreecategories,inwhichtheexaminer:(1i)statedorimpliedthatevidentiaryhaircouldbeassociatedwithaspecificindividualtotheexclusionofallothers;(2ii)assignedtothepositiveassociationastatisticalweightoraprobabilitythattheevidentiaryhairoriginatedfromaparticularsource;or(3iii)citedthenumberof


12

TheimportanceoftheFBI’shairanalysisreviewwasillustratedbythedecisioninJanuary2016byMassachusettsSuperiorCourtJudgeRobertKanetovacatetheconvictionofGeorgePerrot,basedinpartontheFBI’sacknowledgmentoferrorsinhairanalysis.41

ExpandedDOJReview

InMarch2016,DOJannounceditsintentiontoexpandtoadditionalforensicsciencemethodsitsreviewofforensictestimonybytheFBILaboratoryinclosedcriminalcases.Thereviewwillprovidetheopportunitytoassesstheextenttowhichsimilartestimonialoverstatementhasoccurredinotherdisciplines.42DOJplanstolayoutaframeworkforauditingsamplesoftestimonythatcamefromFBIunitshandlingadditionalkindsoffeature-basedevidence,suchastracingtheimpressionsthatgunsleaveonbullets,shoetreads,fibers,soilandothercrime-sceneevidence.

2.4 CognitiveBiasInadditiontotheissuesabove,scientistshavestudiedasubtlerbutequallyimportantproblemthataffectsthereliabilityofconclusionsinmanyfields,includingforensicscience:cognitivebias.Cognitivebiasreferstowaysinwhichhumanperceptionsandjudgmentscanbeshapedbyfactorsotherthanthoserelevanttothedecisionathand.Itincludes“contextualbias,”whereindividualsareinfluencedbyirrelevantbackgroundinformation;“confirmationbias,”whereindividualsinterpretinformation,orlookfornewevidence,inawaythatconformstotheirpre-existingbeliefsorassumptions;and“avoidanceofcognitivedissonance,”whereindividualsarereluctanttoacceptnewinformationthatisinconsistentwiththeirtentativeconclusion.Thebiomedicalsciences,forexample,gotogreatlengthstominimizecognitivebiasbyemployingstrictprotocols,suchasdouble-blindinginclinicaltrials.

Studieshavedemonstratedthatcognitivebiasmayaseriousissueinforensicscience.Forexample,astudybyItielDrorandcolleaguesdemonstratedthatthejudgmentoflatentfingerprintexaminerscanbeinfluencedbyknowledgeaboutotherforensicexaminers’decisions(aformof“confirmationbias”).43ThesestudiesarediscussedinmoredetailinSection5.4.Similarstudieshavereplicatedthesefindingsinotherforensicdomains,includingDNAmixtureinterpretation,microscopichairanalysis,andfireinvestigation.44,45

casesworkedinthelabandthenumberofsuccessfulmatchestosupportaconclusionthatanevidentiaryhairbelongedtoaspecificindividual.Reimer,N.L.“Thehairmicroscopyreviewproject:Anhistoricbreakthroughforlawenforcementandadauntingchallengeforthedefensebar.”TheChampion,(July2013):16.www.nacdl.org/champion.aspx?id=29488.41Commonwealthv.Perrot,407Mass.539(1990).42Hsu,S.S.“JusticeDept.toexpandreviewofFBIforensictechniquesbeyondhairunit.”TheWashingtonPost,February25,2016.www.washingtonpost.com/local/public-safety/justice-dept-to-expand-review-of-fbi-forensic-techniques-beyond-hair-unit/2016/02/25/5adf0b8c-dbd4-11e5-81ae-7491b9b9e7df_story.html.43Dror,I.E.,Charlton,D.,andA.E.Peron.“Contextualinformationrendersexpertsvulnerabletomakingerroneousidentifications.”ForensicScienceInternational,Vol.156(2006):74-8.44See,forexample:Dror,I.E.andG.Hampikian.“SubjectivityandbiasinforensicDNAmixtureinterpretation.”Science&Justice,Vol.51,No.4(2011):204-8;Miller,L.S.“Proceduralbiasinforensicexaminationsofhumanhair.”LawandHumanBehavior,Vol.11(1987):157;andBieber,P.“Fireinvestigationandcognitivebias.”WileyEncyclopediaofForensicScience,2014,availablethroughonlinelibrary.wiley.com/doi/10.1002/9780470061589.fsa1119/abstract.


13

Severalstrategieshavebeenproposedformitigatingcognitivebiasinforensiclaboratories,includingmanagingtheflowofinformationinacrimelaboratorytominimizeexposureoftheforensicanalysttoirrelevantcontextualinformation(suchasconfessionsoreyewitnessidentification)andensuringthatexaminersworkinalinearfashion,documentingtheirfindingaboutevidencefromcrimesciencebeforeperformingcomparisonswithsamplesfromasuspect.46

2.5 StateofForensicScienceThe2009NRCstudyconcludedthatmanyofthesedifficultieswithforensicsciencemaystemfromthehistoricalrealitythatmanymethodsweredevisedasroughheuristicstoaidcriminalinvestigationsandwerenotgroundedinthevalidationpracticesofscientificresearch.47Althoughmanyforensiclaboratoriesdonowrequirenewly-hiredforensicsciencepractitionerstohaveanundergraduatesciencedegree,manypractitionersinforensiclaboratoriesdonothaveadvanceddegreesinascientificdiscipline.48Inaddition,until2015,therewerenoPh.D.programsspecifictoforensicscienceintheUnitedStates(althoughsuchprogramsexistinEurope).49Therehasbeenverylimitedfundingforforensicscienceresearch,especiallytostudythevalidityorreliabilityofthesedisciplines.Seriouspeer-reviewedforensicsciencejournalsfocusedonfeature-comparisonfieldsremainquitelimited.50

Asthe2009NRCstudyandothershavenoted,fundamentally,theforensicsciencesdonotyethaveawell-developed“researchculture.”Importantly,aresearchcultureincludestheprinciplesthat(1)methodsmustbepresumedtobeunreliableuntiltheirfoundationalvalidityhasbeenestablishedbasedonempiricalevidenceand(2)eventhen,scientificquestioningandreviewofmethodsmustcontinueonanongoingbasis.Notably,someforensicpractitionersespousethenotionthatextensive“experience”incaseworkcansubstitutefor

45See,generally,Dror,I.E.“Ahierarchyofexpertperformance.JournalofAppliedResearchinMemoryandCognition.”JournalofAppliedResearchinMemoryandCognition,Vol.5(2016):121-127.46Kassin,S.M.,Dror,I.E.,andJ.Kakucka.“Theforensicconfirmationbias:Problems,perspectives,andproposedsolutions.”JournalofAppliedResearchinMemoryandCognition,Vol.2,No.1(2013):42-52.Seealso:Krane,D.E.,Ford,S.,Gilder,J.,Iman,K.,Jamieson,A.,Taylor,M.S.,andW.C.Thompson.“Sequentialunmasking:AmeansofminimizingobservereffectsinforensicDNAinterpretation.”JournalofForensicSciences,Vol.53,No.4(July2008):1006-7.47NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.128.48NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.223-230.Seealso:Cooney,L.“LatentPrintTrainingtoCompetency:IsitTimeforaUniversalTrainingProgram?”JournalofForensicIdentification,60(2010):223–58(“Theareaswheretherewasnoconsensusincludeddegreerequirements(almosta50/50splitbetweenagenciesthatrequiredafour-yeardegreeorhigherversusthoseagenciesthatrequiredlessthanafour-yeardegreeornodegreeatall)”49NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.223.WhilethereareseveralPh.D.programsincriminaljustice,forensicpsychology,forensicanthropologyorprogramsinchemistryorrelateddisciplinesthatofferaconcentrationinforensicscience,onlySamHoustonStateUniversityCollegeofCriminalJusticeoffersadoctoralprogramin“forensicscience.”See:www.shsu.edu/programs/doctorate-of-philosophy-in-forensic-science.50Mnookin,J.L.,Cole,S.A.,Dror,I.E.,Fisher,B.A.J.,Houck,M.M.,Inman,K.,Kaye,D.H.,Koehler,J.J.,Langenburg,G.,Risinger,D.M.,Rudin,N.,Siegel,J.,andD.A.Stoney.“Theneedforaresearchcultureintheforensicsciences.”UCLALawReview,Vol.725(2011):754-8.

Dr4ensic

Highlight


14

empiricalstudiesofscientificvalidity.51Caseworkisnotscientificallyvalidresearch,andexperiencealonecannotestablishscientificvalidity.Inparticular,onecannotreliablyestimateerrorratesfromcaseworkbecauseonetypicallydoesnothaveindependentknowledgeofthe“groundtruth”or“rightanswer.”52

Beyondthefoundationalissueofscientificvalidity,mostfeature-comparisonfieldshistoricallygaveinsufficientattentiontotheimportanceof“blinding”practitionerstopotentiallybiasinginformation;developingobjectivemeasuresofassessmentandinterpretation;payingcarefulattentiontoerrorratesandtheirmeasurement;anddevelopingobjectiveassessmentsofthemeaningofanassociationbetweenasampleanditspotentialsource.53

The2009NRCreportstimulatedsomeintheforensicsciencecommunitytorecognizetheseflaws.Someforensicscientistshaveembracedtheneedtoplaceforensicsonasolidscientificfoundationandhaveundertakeninitialeffortstodoso.54

2.6 StateofForensicPracticeInvestigationsofforensicpracticehavelikewiseunearthedproblemsstemmingfromthelackofastrong“qualityculture.”Specifically,dozensofinvestigationsofcrimelaboratories—primarilyatthestateandlocallevel—haverevealedrepeatedfailuresconcerningthehandlingandprocessingofevidenceandincorrectinterpretationofforensicanalysisresults.55

51SeeSection4.7.52Ibid.53NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):pp.8;124;184-5;188-91.SeealsoKoppl,R.andKrane,D.Minimizingandleveragingbiasinforensicscience.InRobertsonCT,KesselheimAS,editors.Blindingasasolutiontobias:Strengtheningbiomedicalscience,forensicscience,andlaw.Atlanta,GA:Elsevier;2016.54SeeSection4.8.55Afewexamplesofsuchinvestigationsinclude:(1)a2-yearindependentinvestigationoftheHoustonPoliceDepartment’scrimelabthatresultedinthereviewof3,500cases(FinalReportoftheIndependentInvestigatorfortheHoustonPoliceDepartmentCrimeLaboratoryandPropertyRoom,PreparedbyMichaelR.Bromwich,June13,2007.Availableat:http://www.hpdlabinvestigation.org/reports/070613report.pdf);(2)theinvestigationandclosureoftheDetroitpolicecrimelab’sfirearmsunitfollowingthediscoveryofevidencecontaminationandfailuretoproperlymaintaintestingequipment(seeNickBunkley,“Detroitpolicelabisclosedafterauditfindsseriouserrorsinmanycases,”NewYorkTimes,September25,2008,availableathttp://www.nytimes.com/2008/09/26/us/26detroit.html?_r=0);(3)a2010investigationofNorthCarolina’sStateBureauofInvestigationcrimelaboratorythatfoundthatagentsconsistentlywithholdexculpatoryevidenceordistortedevidenceinmorethan230casesovera16yearperiod(seeChrisSweckerandMichaelWolf,AnIndependentReviewoftheSBIForensicLaboratory4(2010));and(4)a2013reviewoftheNewYorkCitymedicalexaminer’sofficehandlingofDNAevidenceinmorethan800rapecases(seeStateofNewYork,OfficeoftheInspectorGeneral.December2013,availableat:https://www.ig.ny.gov/sites/default/files/pdfs/OCMEFinalReport.pdf).Oneanalysisestimatedthatatleastfiftymajorlaboratoriesreportedfraudbyanalysts,evidencedestruction,failedproficiencytests,misrepresentingfindingsintestimony,ortamperingwithdrugsbetween2005and2011.Twenty-eightoftheselabswerenationallyaccredited.MemorandumfromMarvinSchechtertoNewYorkStateCommissiononForensicScience(March25,2011):pp.243-4.www.americanbar.org/content/dam/aba/administrative/legal_aid_indigent_defendants/ls_sclaid_def_train_memo_schechter.authcheckdam.pdf.

Dr4ensic

Highlight


15

Variouscommentatorshavepointedoutafundamentalissuethatmayunderlietheseseriousproblems:thefactthatnearlyallcrimelaboratoriesarecloselytiedtotheprosecutionincriminalcases.Thisstructureunderminesthegreaterobjectivitytypicallyfoundintestinglaboratoriesinotherfieldsandcreatessituationswherepersonnelmaymakeerrorsduetosubtlecognitivebiasorovertpressure.56

The2009NRCreport(citedearlieranddiscussedinmoredetailinthenextsection)recommendedthatallpublicforensiclaboratoriesandfacilitiesberemovedfromtheadministrativecontroloflawenforcementagenciesorprosecutors’offices.57Forexample,Houston—afterdisbandingitscrimelaboratorytwiceinthreeyears—followedthisrecommendationand,despitesignificantpoliticalpushback,succeededintransitioningthelaboratoryintoanindependentforensicsciencecenter.58

2.7 NationalResearchCouncilReportThe2009NRCreport,StrengtheningForensicScienceintheUnitedStates:APathForward,wasthemostcomprehensivereviewtodateoftheforensicsciencesintheUnitedStates.Thereportmadeclearthatthetypesofproblems,irregularities,andmiscarriagesofjusticeoutlinedinthisreportcannotsimplybeattributedtoahandfulofrogueanalystsorunderperforminglaboratories.Instead,thereportfoundtheproblemsplaguingtheforensicsciencecommunityaresystemicandpervasive—theresultoffactorsincludingahighdegreeoffragmentation(includingdisparateandofteninadequatetrainingandeducationalrequirements,resources,andcapacitiesoflaboratories),alackofstandardizationofthedisciplines,insufficienthigh-qualityresearchandeducation,andadearthofpeer-reviewedstudiesestablishingthescientificbasisandvalidityofmanyroutinelyusedforensicmethods.

Shortcomingsintheforensicscienceswereespeciallyprevalentamongthefeature-comparisondisciplines.The2009NRCreportfoundthatmanyofthesedisciplineslackedwell-definedsystemsfordeterminingerrorratesandhadnotdonestudiestoestablishtheuniquenessorrelativerarityorcommonalityoftheparticularmarksorfeaturesexamined.Inaddition,proficiencytesting,whereithadbeenconducted,showedinstancesofpoorperformancebyspecificexaminers.Inshort,thereportconcludedthat“muchforensicevidence—including,forexample,bitemarksandfirearmandtoolmarkidentifications—isintroducedincriminaltrialswithoutany

56The2009NRCReport(pp.24-5)states,“Thebestscienceisconductedinascientificsettingasopposedtoalawenforcementsetting.Becauseforensicscientistsoftenaredrivenintheirworkbyaneedtoansweraparticularquestionrelatedtotheissuesofaparticularcase,theysometimesfacepressuretosacrificeappropriatemethodologyforthesakeofexpediency.”Seealso:Giannelli,P.G.“Independentcrimelaboratories:Theproblemofmotivationalandcognitivebias.”UtahLawReview,(2010):247-66andThompson,S.G.CopsinLabCoats:CurbingWrongfulConvictionsthroughIndependentForensicLaboratories.CarolinaAcademicPress(2015).57NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):Recommendation4,p.24.58TheHoustonForensicScienceCenteropenedinApril2014,replacingtheformerHoustonPoliceDepartmentCrimeLaboratory.TheCenteroperatesasa“localgovernmentcorporation”withitsowndirectors,officers,andemployees.ThestructurewasintentionallydesignedtoinsulatetheCenterfromundueinfluencebypolice,prosecutors,electedofficials,orspecialinterestgroups.See:Thompson,S.G.CopsinLabCoats:CurbingWrongfulConvictionsthroughIndependentForensicLaboratories.CarolinaAcademicPress(2015).:p.214.


16

meaningfulscientificvalidation,determinationoferrorrates,orreliabilitytestingtoexplainthelimitsofthediscipline.”59

The2009NRCreportfoundthattheproblemsplaguingtheforensicsciencesweresoseverethattheycouldonlybeaddressedby“anationalcommitmenttooverhaulthecurrentstructurethatsupportstheforensicsciencecommunityinthiscountry.”60Underlyingthereport’s13corerecommendationswasacallforleadershipatthehighestlevelsofbothFederalandStateGovernmentsandthepromotionandadoptionofalong-termagendatopulltheforensicscienceenterpriseupfromitscurrentweaknesses.

The2009NRCreportcalledforstudiestotestwhethervariousforensicmethodsarefoundationallyvalid,includingperformingempiricaltestsoftheaccuracyoftheresults.Italsocalledforthecreationofanew,independentFederalagencytoprovideneededoversightoftheforensicsciencesystem;standardizationofterminologyusedinreportingandtestifyingabouttheresultsofforensicsciences;theremovalofpublicforensiclaboratoriesfromtheadministrativecontroloflawenforcementagencies;implementationofmandatorycertificationrequirementsforpractitionersandmandatoryaccreditationprogramsforlaboratories;researchonhumanobserverbiasandsourcesofhumanerrorinforensicexaminations;thedevelopmentoftoolsforadvancingmeasurement,validation,reliability,andproficiencytestinginforensicscience;andthestrengtheninganddevelopmentofgraduateandcontinuouseducationandtrainingprograms.

2.8 RecentProgressInresponsetothe2009NRCreport,theObamaAdministrationinitiatedaseriesofreformeffortsaimedatstrengtheningtheforensicsciences,beginningwiththecreationin2009ofaSubcommitteeonForensicScienceoftheNationalScienceandTechnologyCouncil’sCommitteeonSciencethatwaschargedwithconsideringhowbesttoachievethegoalsofthereport.Theresultingactivitiesaredescribedinsomedetailbelow.

NationalCommissiononForensicScience

In2013,theDOJandNIST,withsupportfromtheWhiteHouse,signedaMemorandumofUnderstandingthatoutlinedaframeworkforcooperationandcollaborationbetweenthetwoagenciesinsupportofeffortstostrengthenforensicscience.

In2013,DOJestablishedaNationalCommissiononForensicScience(NCFS),aFederaladvisorycommitteereportingtotheAttorneyGeneral.Co-chairedbytheDeputyAttorneyGeneralandtheDirectorofNIST,theNCFS’s32membersincludesevenacademicscientistsandfiveothersciencePhDs;theothermembersincludejudges,attorneysandforensicpractitioners.ItischargedwithprovidingpolicyrecommendationstotheAttorneyGeneral.61TheNCFSissuesformalrecommendationstotheAttorneyGeneral,aswellas“views

59NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):pp.107-8.60NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):pp.xx.61See:www.justice.gov/ncfs.


17

documents”thatreflecttwo-thirdsmajorityviewofNCFSbutdonotrequestspecificactionbytheAttorneyGeneral.Todate,theNCFShasissuedtenrecommendationsconcerning,amongotherthings,accreditationofforensiclaboratoriesandcertificationofforensicpractitioners,advancingtheinteroperabilityoffingerprintinformationsystems,developmentofrootcauseanalysisprotocolsforforensicserviceproviders,andenhancingcommunicationsamongmedical-examinerandcoroneroffices.62Todate,theAttorneyGeneralhasformallyadoptedthefirstsetofrecommendationsonaccreditation,63andhasdirectedtheDepartmenttobegintotakestepstowardaddressingsomeoftheotherrecommendationsputforwardtodate.64

In2014,NISTestablishedtheOrganizationofScientificAreaCommittees(OSAC),acollaborativebodyofmorethan600volunteermemberslargelydrawnfromtheforensicsciencecommunity.65OSACwasestablishedtosupportthedevelopmentofvoluntarystandardsandguidelinesforconsiderationbytheforensicpractitionercommunity.66ThestructureconsistsofsixScientificAreaCommittees(SACs)and25subcommitteesthatworktodevelopstandards,guidelines,andcodesofpracticeforeachoftheforensicsciencedisciplinesandmethodologies.67Threeoverarchingresourcecommitteesprovideguidanceonquestionsoflaw,work-flowprocesses,andqualityassurance.AlldocumentsdevelopedbytheSACsareapprovedbyaForensicScienceStandardsBoard(FSSB),acomponentoftheOSACstructure,forlistingontheOSACRegistryofApprovedStandards.OSACisnotaFederaladvisorycommittee.

FederalFundingOfResearch

TheFederalgovernmenthasalsotakenstepstoaddressonefactorcontributingtotheproblemswithforensicscience—thelackofarobustandrigorousscientificresearchcommunityinmanydisciplinesinforensicscience.Whiletherearemultiplereasonsfortheabsenceofsucharesearchcommunity,onereasonisthat,unlikemostscientificdisciplines,therehasbeentoolittlefundingtoattractandsustainasubstantialcadreofexcellentscientistsfocusedonfundamentalresearchinforensicscience.

TheNationalScienceFoundation(NSF)hasrecentlybeguneffortstohelpaddressthisfoundationalshortcomingofforensicscience.In2013,NSFsignaleditsinterestinthisareaandencouragedresearcherstosubmitresearchproposalsaddressingfundamentalquestionsthatmightadvanceknowledgeandeducationintheforensic

62ForafulllistofdocumentsapprovedbyNCFS,seewww.justice.gov/ncfs/work-products-adopted-commission.63DepartmentofJustice.“JusticeDepartmentannouncesnewaccreditationpoliciestoadvanceforensicscience.”(December7,2015,pressrelease).www.justice.gov/opa/pr/justice-department-announces-new-accreditation-policies-advance-forensic-science.64MemorandumfromtheAttorneyGeneraltoHeadsofDepartmentComponentsRegardingRecommendationsoftheNationalCommissiononForensicScience,March17,2016.www.justice.gov/ncfs/file/841861/download.65Membersincludeforensicsciencepractitionersandotherexpertswhorepresentlocal,state,andfederalagencies;academia;andindustry.66Formoreinformationsee:www.nist.gov/forensics/osac.cfm.67ThesixScientificAreaCommitteesunderOSACare:Biology/DNA,Chemistry/InstrumentalAnalysis,CrimeScene/DeathInvestigation,Digital/Multimedia,andPhysics/PatternInterpretation(www.nist.gov/forensics/upload/OSAC-Block-Org-Chart-3-17-2015.pdf).


18

sciences.68AsaresultofaninteragencyprocessledbyOSTPandNSF,incollaborationwiththeNationalInstituteofJustice(NIJ),invitedproposalsforthecreationofnew,multi-disciplinaryresearchcentersforfundingin2014.69Basedonourreviewofgrantabstracts,PCASTestimatesthatNSFcommitsatotalofapproximately$4.5millionperyearinsupportforextramuralresearchprojectsonfoundationalforensicscience.

NISThasalsotakenstepstoaddressthisissue,bycreatinganewForensicScienceCenterofExcellence,calledtheCenterforStatisticsandApplicationsinForensicEvidence(CSAFE),thatwillfocusitsresearcheffortsonimprovingthestatisticalfoundationforlatentprints,ballistics,tiremarks,handwriting,bloodstainpatterns,toolmarks,patternevidenceanalyses,andforcomputerandinformationsystems,mobiledevices,networktraffic,socialmedia,andGPSdigitalevidenceanalyses.70CSFAEisfundedunderacooperativeagreementwithIowaStateUniversity,tosetupacenterinpartnershipwithinvestigatorsatCarnegieMellonUniversity,theUniversityofVirginia,andtheUniversityofCalifornia,Irvine;thetotalsupportis$20millionoverfiveyears.PCASTestimatesthatNISTcommitsatotalofapproximately$5millionperyearinsupportforextramuralresearchprojectsonfoundationalforensicscience,consistingofapproximately$4milliontoCSAFEandapproximately$1milliontootherprojects.

NIJhasnobudgetallocatedspecificallyforforensicscienceresearch.Inordertosupportresearchactivities,NIJmustdrawfromitsbasefunding,fundingfromtheOfficeofJusticePrograms’(OJP)assistanceprogramsforresearchandstatistics,orfromtheDNAbacklogreductionprograms.71Mostofitsresearchsupportisdirectedtoappliedresearch.AlthoughitisdifficulttoclassifyNIJ’sresearchprojects,weestimatethatNIJcommitsatotalofapproximately$4millionperyeartosupportextramuralresearchprojectsonfundamentalforensicscience.72

Evenwiththerecentincreases,thetotalextramuralfundingforfundamentalresearchinforensicscienceacrossNSF,NIST,andNIJisthuslikelytobeintherangeofonly$13.5millionperyear.

68See:DearColleagueLetter:ForensicScience–OpportunityforBreakthroughsinFundamentalandBasicResearchandEducation.www.nsf.gov/pubs/2013/nsf13120/nsf13120.jsp.69ThecentersNSFisproposingtocreateareIndustry/UniversityCooperativeResearchCenters(I/UCRCs).I/UCRCsarecollaborativebydesignandcouldbeeffectiveinhelpingtobridgethescientificandculturalgapbetweenacademicresearcherswhoworkinforensics-relevantfieldsofscienceandforensicpractitioners.www.nsf.gov/pubs/2014/nsf14066/nsf14066.pdf.70NationalInstituteofStandardsandTechnology.“NewNISTCenterofExcellencetoImproveStatisticalAnalysisofForensicEvidence.”(2015).www.nist.gov/forensics/center-excellence-forensic052615.cfm.71NationalAcademiesofSciences,Engineering,andMedicine.SupportforForensicScienceResearch:ImprovingtheScientificRoleoftheNationalInstituteofJustice.TheNationalAcademiesPress.WashingtonDC.(2015).Accordingtothereport,“CongressionalappropriationstosupportNIJ’sresearchprogramsdeclinedduringtheearlytomid-2000sandremaininsufficient,especiallyinlightofthegrowingchallengesfacingtheforensicsciencecommunity…Withlimitedbasefunding,NIJfundsresearchanddevelopmentfromtheappropriationsforDNAbacklogreductionprogramsandotherassistanceprograms.Thesecarved-outfundsareessentiallysupportingNIJ’scurrentforensicscienceportfolio,buttherearepressurestolimittheamountusedforresearchfromtheseprograms.Inthepast3years,fundingfortheseassistanceprogramshasdeclined;therefore,fundsavailableforresearchhavealsobeenreduced.”72U.S.DepartmentofJustice,NationalInstituteofJustice.“ReportForensicScience:FiscalYear2015FundingforDNAAnalysis,CapacityEnhancementandOtherForensicActivities.”2016.


19

The2009NRCreportfoundthat

Forensicscienceresearchis[overall]notwellsupported....Relativetootherareasofscience,theforensicsciencedisciplineshaveextremelylimitedopportunitiesforresearchfunding.AlthoughtheFBIandNIJhavesupportedsomeresearchintheforensicsciencedisciplines,thelevelofsupporthasbeenwellshortofwhatisnecessaryfortheforensicsciencecommunitytoestablishstronglinkswithabroadbaseofresearchuniversitiesandthenationalresearchcommunity.Moreover,fundingforacademicresearchislimited...,whichcaninhibitthepursuitofmorefundamentalscientificquestionsessentialtoestablishingthefoundationofforensicscience.Finally,thebroaderresearchcommunitygenerallyisnotengagedinconductingresearchrelevanttoadvancingtheforensicsciencedisciplines.73

Thecourtfoundthatthesystolictesthad“notyetgainedsuchstandingandscientificrecognitionamongphysiologicalandpsychologicalauthorities,”andwasthereforeinadmissible.

A2015NRCreport,SupportforForensicScienceResearch:ImprovingtheScientificRoleoftheNationalInstituteofJustice,foundthatthestatusofforensicscienceresearchfundinghasnotimprovedmuchsincethe2009report.74

Inaddition,theDefenseForensicScienceCenter(DFSC)hasrecentlybeguntosupportextramuralresearchspanningtheforensicsciencedisciplinesaspartofitsmissiontoprovidespecializedforensicandbiometricresearchcapabilitiesandsupporttotheDepartmentofDefense.RedesignatedasDFSCin2013,theCenterwasformerlytheU.S.ArmyCriminalInvestigationLaboratory(USACIL),originallychargedwithsupportingcriminalinvestigationswithinthemilitarybutadditionallytaskedin2007withprovidingan“enduringexpeditionaryforensicscapability”,inresponseinparttotheneedtoinvestigateandprosecuteexplosivesattacksinIraqandAfghanistan.WhilethebulkofDFSCsupporthastraditionallysupportedresearchinDNAanalysisandbiochemistry,theCenterhasrecentlydirectedresourcestowardprojectstoaddresscriticalfoundationalgapsinotherdisciplines,includingfirearmsandlatentprintanalysis.

Notably,DFSChashelpedstimulateresearchintheforensicsciencecommunity.DiscussionsbetweenDFSCandtheAmericanSocietyofCrimeLabDirectors(ASCLD)ledASCLDtobeginhostingmeetingstoidentifyresearchprioritiesfortheforensicsciencecommunity.DFSCagreedtofundtwofoundationalstudiestoaddressthehighestpriorityresearchneedsidentifiedbytheForensicResearchCommitteeoftheAmericanSocietyofCrimeLabDirectors(ASCLD):thefirstindependenterrorratestudyonfirearmsanalysisandaDNAmixtureinterpretationstudy(seeChapter5).Seventy-fivepercentofthetotalfundingsupportedprojectswithregardtoDNA/biochemistry;9%digitalevidence;8%non-DNApatternevidence;and8%chemistry.75AsisthecaseforNIJ,thereisnolineiteminDFSC’sbudgetdedicatedtoforensicscienceresearch;DFSCinsteadmustsolicitfundingfrommultiplesourceswithinDODtosupportthisresearch.

73NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.78.74NationalAcademiesofSciences,Engineering,andMedicine.SupportforForensicScienceResearch:ImprovingtheScientificRoleoftheNationalInstituteofJustice.TheNationalAcademiesPress.WashingtonDC.(2015):p.15.75DefenseForensicScienceCenter,OfficeoftheChiefScientist,AnnualResearchPortfolioReport,January5,2016.


20

ACriticalGap:ScientificValidity

TheAdministrationhastakenimportantandmuchneededinitialstepsbycreatingmechanismstodiscusspolicy,developbestpracticesforpractitionersofspecificmethods,andsupportscientificresearchintothevalidityofmethods.Atthesametime,worktodatehasnotaddressedthe2009NRCreport’scalltoexaminethefundamentalscientificvalidityandreliabilityofmanyforensicmethodsusedeverydayincourts.Theremainderofourreportfocusesonthatissue.


21

3. TheRoleofScientificValidityintheCourts

Thecentralfocusofthisreportisthescientificvalidityofforensicevidence—morespecifically,forensicevidencerelatingtothecomparisonoffeatures(in,forexample,DNA,latentfingerprints,bulletmarksandotheritems).Thereliabilityofevidenceandmethodsforinterpretingitisafundamentalconsiderationthroughoutscience.Accordingly,everyscientificfieldhasawell-developed,domain-specificunderstandingofwhatscientificvalidityentails.

Theconceptofscientificvalidityalsoplaysanimportantroleinthelegalsystem.Inparticular,asnotedinChapter1,theFederalRulesofEvidencerequirethatexperttestimonyaboutforensicsciencemustbetheproductof“reliableprinciplesandmethods”thathavebeen“reliablyapplied...tothefactsofthecase.”

Thisreportexplicatesthescientificcriteriaforscientificvalidityinthecaseofforensicfeature-comparisondisciplines,forusebothwithinthelegalsystemandbythoseworkingtostrengthenthescientificunderpinningsofthosedisciplines.Beforedelvingintothatscientificexplication,weprovideinthischapteraverybriefsummary,aimedprincipallyatscientistandlayreaders,oftherelevantlegalbackgroundandterms,aswellasthenatureofthisintersectionbetweenlawandscience.

3.1 EvolutionofAdmissibilityStandardsOverthecourseofthe20thcentury,thelegalsystem’sapproachfordeterminingtheadmissibilityofscientificevidencehasevolvedinresponsetoadvancesinscience.In1923,inFryev.UnitedStates,76theCourtofAppealsfortheDistrictofColumbiaconsideredtheadmissibilityoftestimonyconcerningresultsofapurported“liedetector”,asystolic-blood-pressuredeceptiontestthatwasaprecursortothepolygraphmachine.Afterdescribingthedeviceanditsoperation,theCourtrejectedthetestimony,stating:

[W]hilecourtswillgoalongwayinadmittingexperttestimonydeducedfromawell-recognizedscientificprincipleordiscovery,thethingfromwhichthedeductionismademustbesufficientlyestablishedtohavegainedgeneralacceptanceintheparticularfieldinwhichitbelongs.77

Thecourtfoundthatthesystolictesthad“notyetgainedsuchstandingandscientificrecognitionamongphysiologicalandpsychologicalauthorities,”andwasthereforeinadmissible.

Morethanahalf-centurylater,theFederalRulesofEvidencewereenactedintolawin1975toguidecriminalandcivillitigationinFederalcourts.Rule702,initsoriginalform,statedthat:

76Fryev.UnitedStates,293F.1013(D.C.Cir.1923).77Ibid.,at1014.


22

Ifscientific,technical,orotherspecializedknowledgewillassistthetrieroffacttounderstandtheevidenceortodetermineafactinissue,awitnessqualifiedasanexpertbyknowledge,skill,experience,training,oreducation,maytestifytheretointheformofanopinionorotherwise.78

Therewasconsiderabledebateamonglitigants,judges,andlegalscholarsastowhethertheruleembracedtheFryestandardorestablishedanewstandard.79In1993,theUnitedStatesSupremeCourtsoughttoresolvethesequestionsinitslandmarkrulinginDaubertv.MerrellDowPharmaceuticals.IninterpretingRule702,theDaubertCourtheldthattheFederalRulesofEvidencesupersededFryeasthestandardforadmissibilityofexpertevidenceinFederalcourts.TheCourtrejected“generalacceptance”asthestandardforadmissibilityandinsteadheldthattheadmissibilityofscientificexperttestimonydependedonitsscientificreliability.

WhereFryetoldjudgestodefertothejudgmentoftherelevantexpertcommunity,Daubertassignedtrialcourtjudgestheroleof“gatekeepers”chargedwithensuringthatexperttestimony“restsonreliablefoundation.”80

TheCourtstatedthat“thetrialjudgemustdetermine...whetherthereasoningormethodologyunderlyingthetestimonyisscientificallyvalid.”81Itidentifiedfivefactorsthatajudgeshould,amongothers,ordinarilyconsiderinevaluatingthevalidityofanunderlyingmethodology.Thesefactorsare:(1)whetherthetheoryortechniquecanbe(andhasbeen)tested;(2)whetherthetheoryortechniquehasbeensubjectedtopeerreviewandpublication;(3)theknownorpotentialrateoferrorofaparticularscientifictechnique;(4)theexistenceandmaintenanceofstandardscontrollingthetechnique’soperation;and(5)ascientifictechnique’sdegreeofacceptancewithinarelevantscientificcommunity.

TheDaubertcourtalsonotedthatjudgesevaluatingproffersofexpertscientifictestimonyshouldbemindfulofotherapplicablerules,including:

• Rule403,whichpermitstheexclusionofrelevantevidence“ifitsprobativevalueissubstantiallyoutweighedbythedangerofunfairprejudice,confusionoftheissues,ormisleadingthejury…”(notingthatexpertevidencecanbe“bothpowerfulandquitemisleadingbecauseofthedifficultyinevaluatingit.”);and

• Rule706,whichallowsthecourtatitsdiscretiontoprocuretheassistanceofanexpertofitsownchoosing.82

78ActofJanuary2,1975,Pub.LawNo.93-595,88Stat.1926(1975).See:federalevidence.com/pdf/FRE_Amendments/1975_Orig_Enact/1975-Pub.L._93-595_FRE.pdf.79See:Giannelli,P.C.“Theadmissibilityofnovelscientificevidence:Fryev.UnitedStates,ahalf-centurylater.”ColumbusLawReview,Vol.80,No.6(1980);McCabe,J.“DNAfingerprinting:ThefailingsofFrye,”NortherIllinoisUniversityLawReview,Vol.16(1996):455-82;andPage,M.,Taylor,J.,andM.Blenkin.“ForensicidentificationscienceevidencesinceDaubert:PartII—judicialreasoningindecisionstoexcludeforensicidentificationevidenceongroundsofreliability.”JournalofForensicSciences,Vol.56,No.4(2011):913-7.80Daubert,at597.81Daubert,at580.Seealso,FN9.82Daubert,at595,citingWeinstein,138F.R.D.,at632.


23

CongressamendedRule702in2000tomakeitmoreprecise,andmadefurtherstylisticchangesin2011.Initscurrentform,Rule702imposesfourrequirements:

Awitnesswhoisqualifiedasanexpertbyknowledge,skill,experience,training,oreducationmaytestifyintheformofanopinionorotherwiseif:(a)theexpert’sscientific,technical,orotherspecializedknowledgewillhelpthetrieroffacttounderstandtheevidenceortodetermineafactinissue;(b)thetestimonyisbasedonsufficientfactsordata;(c)thetestimonyistheproductofreliableprinciplesandmethods;and(d)theexperthasreliablyappliedtheprinciplesandmethodstothefactsofthecase.

AnAdvisoryCommittee’sNotetoRule702alsospecifiedanumberofreliabilityfactorsthatsupplementthefivefactorsenumeratedinDaubert.Amongthosefactorsis“whetherthefieldofexpertiseclaimedbytheexpertisknowntoreachreliableresults.”83,84

Manystateshaveadoptedrulesofevidencethattrackkeyaspectsofthesefederalrules.Suchrulesarenowthelawinoverhalfofthestates,whileotherstatescontinuetofollowtheFryestandardorvariationsofit.85

3.2 FoundationalValidityandValidityasAppliedAsdescribedinDaubert,thelegalsystemenvisionsanimportantconversationbetweenlawandscience:

“The[judge’s]inquiryenvisionedbyRule702is,weemphasize,aflexibleone.Itsoverarchingsubjectisthescientificvalidity—andthustheevidentiaryrelevanceandreliability—oftheprinciplesthatunderlieaproposedsubmission.”86

Legalandscientificconsiderationsthusbothplayimportantroles.

83See:Fed.R.Evid.702AdvisoryCommitteenote(2000).ThefollowingfactorsmayberelevantunderRule702:whethertheunderlyingresearchwasconductedindependentlyoflitigation;whethertheexpertunjustifiablyextrapolatedfromanacceptedpremisetoanunfoundedconclusion;whethertheexperthasadequatelyaccountedforobviousalternativeexplanations;whethertheexpertwasascarefulasshewouldbeinherprofessionalworkoutsideofpaidlitigation;andwhetherthefieldofexpertiseclaimedbytheexpertisknowntoreachreliableresults[emphasisadded].84Thisnotehasbeenpointedtoassupportforeffortstochallengeentirefieldsofforensicscience,includingfingerprintsandhaircomparisons.See:Giannelli,P.C.“TheSupremeCourt’s‘Criminal’DaubertCases.”SetonHallLawReview,Vol.33(2003):1096.85EvenundertheFryeformulation,theviewsofscientistsaboutthemeaningofreliabilityarerelevant.Fryerequiresthatascientifictechniqueormethodmust“havegeneralacceptance”intherelevantscientificcommunitytobeadmissible.Asascientificmatter,therelevantscientificcommunityforassessingthereliabilityoffeature-comparisonsciencesincludesmetrologists(includingstatisticians)aswellasotherphysicalandlifescientistsfromdisciplinesonwhichthespecificmethodsarebased.Importantly,thecommunityisnotlimitedtoforensicscientistswhopracticethespecificmethod.Forexample,theFryecourtevaluatedwhethertheprofferedliedetectorhadgained“standingandscientificrecognitionamongphysiologicalandpsychologicalauthorities,”ratherthanamongliedetectorexperts.Fryev.UnitedStates,293F.1013(D.C.Cir.1923).86Daubert,at594


24

(1)Theadmissibilityofexperttestimonydependsonathresholdtestof,amongotherthings,whetheritmeetscertainlegalstandardsembodiedinRule702.Thesedecisionsaboutadmissibilityareexclusivelytheprovinceofthecourts.

(2)Yet,asnotedabove,theoverarchingsubjectofthejudge’sinquiryunderRule702is“scientificvalidity.”Itistheproperprovinceofthescientificcommunitytoprovideguidanceconcerningscientificstandardsforscientificvalidity.

PCASTdoesnotopinehereonthelegalstandards,butseeksonlytoclarifythescientificstandardsthatunderliethem.Forcompleteclarityaboutourintent,wehaveadoptedspecifictermstorefertothescientificstandardsfortwokeytypesofscientificvalidity,whichwemeantocorrespond,asscientificstandards,tothelegalstandardsinRule702(c,d)):

(1)By“foundationalvalidity”,wemeanthescientificstandardforwhetherevidenceisbasedon“reliableprinciplesandmethods”,and

(2)By“validityasapplied”,wemeanthescientificstandardforwhetherone“hasreliablyappliedtheprinciplesandmethods.”

Inthenextchapter,weturntodiscussingthescientificstandardsfortheseconcepts.Weclosethischapterbynotingthatansweringthequestionofscientificvalidityintheforensicdisciplinesisimportantnotjustforthecourtsbutalsobecauseitsetsqualitystandardsthatrippleoutthroughoutthesedisciplines—affectingpracticeanddefiningnecessaryresearch.


25

4. ScientificCriteriaforValidityandReliability

ofForensicFeature-ComparisonMethodsInthisreport,PCASThaschosentofocusondefiningthevalidityandreliabilityofonespecificareawithinforensicscience:forensicfeature-comparisonmethods.Wehavedonesobecauseitisbothpossibleandimportanttodosoforthisparticularclassofmethods.

• Itispossiblebecausefeaturecomparisonisacommonscientificactivity,andsciencehasclearstandardsfordeterminingwhethersuchmethodsarereliable.Inparticular,feature-comparisonmethodsbelongsquarelytothedisciplineofmetrology—thescienceofmeasurementanditsapplication.87,88

• Itisimportantbecauseithasbecomeapparent,overthepastdecade,thatfaultyforensicfeaturecomparisonhasledtonumerousmiscarriagesofjustice.89Ithasalsobeenrevealedthattheproblemsarenotduesimplytopoorperformancebyafewpractitioners,butrathertothefactthatthereliabilityofmanyforensicfeature-comparisonmethodshasneverbeenmeaningfullyevaluated.90

87InternationalVocabularyofMetrology–BasicandGeneralConceptsandAssociatedTerms(VIM3rdedition)JCGM200:2012.88Thatforensicfeature-comparisonmethodsbelongtothefieldofmetrologyisclearfromthefactthatNIST—whosemissionistoassisttheNationby“advancingmeasurementscience,standardsandtechnology,”andwhichistheworld’sleadingmetrologicallaboratory—isthehomewithintheFederalgovernmentforresearcheffortsonforensicscience.NIST’sprogramsincludeinternalresearch,extramuralresearchfunding,conferences,andpreparationofreferencematerialsandstandards.http://www.nist.gov/public_affairs/mission.cfm;andhttp://www.nist.gov/forensics/index.cfm.Forensicfeature-comparisonmethodsinvolvedeterminingwhethertwosetsoffeaturesagreewithinagivenmeasurementtolerance.89DNA-basedre-examinationofpastcaseshasledsofartotheexonerationsof342defendants,including20whohadbeensentencedtodeath,andtotheidentificationof147realperpetrators.SeeInnocenceProject,“DNAExonerationsintheUnitedStates.”See:www.innocenceproject.org/dna-exonerations-in-the-united-states.Reviewsofthesecaseshaverevealedthatroughlyhalfreliedinpartonexperttestimonythatwasbasedonmethodsthathadnotbeensubjectedtomeaningfulscientificscrutinyorthatincludedscientificallyinvalidclaimsofaccuracy.See:Gross,S.R.,andM.Shaffer.“ExonerationsintheUnitedStates,1989-2012.”NationalRegistryofExonerations,(2012)availableat:www.law.umich.edu/special/exoneration/Documents/exonerations_us_1989_2012_full_report.pdf;Garrett,B.L.,andP.J.Neufeld.“Invalidforensicsciencetestimonyandwrongfulconvictions.”VirginiaLawReview,Vol.91,No.1(2009):1-97;NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):pp.42-3.ExampleincludeindividualswhowereexoneratedbasedonDNAevidenceafterhavingbeenconvictedof:rape,basedonhairanalysis(15yearsserved);childmurder,basedonbitemarkanalysis(twocases,bothmorethan13yearsserved);murderbasedonhairanalysisof13hairs,whichnoneofwhich,DNAanalysislatershowed,camefromthedefendantandonecamefromadog(20years);shootingapoliceofficer,basedonfingerprintanalysis(5years);andrapeandmurder,basedonfiberandhairanalysis(20years).90SeeChapter5.


26

Comparedtomanytypesofexperttestimony,testimonybasedonforensicfeature-comparisonmethodsposesuniquedangersofmisleadingjurorsfortworeasons:

• Thevastmajorityofjurorshavenoindependentabilitytointerprettheprobativevalueofresultsbasedonthedetection,comparisonandfrequencyofscientificevidence.Ifmatchinghalvesofaransomnotewerefoundatacrimesceneandatadefendant’shome,jurorscouldrelyontheirownexperiencestoassesshowunlikelyitisthattwotornscrapswouldmatchiftheywerenotinfactfromasingleoriginalnote.Ifawitnessweretodescribeaperpetratoras“tallandbushyhaired,”jurorscouldmakeareasonablejudgmentofhowmanypeoplemightmatchthedescription.But,ifanexpertwitnessweretosaythat,intwoDNAsamples,thethirdexonoftheDYNC1H1geneisprecisely174nucleotidesinlength,mostjurorswouldhavenowaytoknowiftheyshouldbeimpressedbythecoincidence;theywouldbecompletelydependentonexpertstatementsgarbedinthemantleofscience.(Asithappens,theyshouldnotbeimpressedbytheprecedingstatement:AttheDNAlocuscited,morethan99.9%ofpeoplehaveafragmentoftheindicatedsize.91)

• Thepotentialprejudicialimpactisunusuallyhigh,becausejurorsarelikelytooverestimatetheprobativevalueofa“match”betweensamples.Indeed,theDOJitselfhistoricallyoverestimatedtheprobativevalueofmatchesinitslongstandingcontention,nowacknowledgedtobeinappropriate,thatlatentfingerprintanalysiswas“infallible.”92Similarly,aformerheadoftheFBI’sfingerprintunittestifiedthattheFBIhad“anerrorrateofoneperevery11millioncases”.93Inanonlineexperiment,researchersaskedmockjurorstoestimatethefrequencythataqualified,experiencedforensicscientistwouldmistakenlyconcludethattwosamplesofspecifiedtypescamefromthesamepersonwhentheyactuallycamefromtwodifferentpeople.Themockjurorsbelievedsucherrorsarelikelytooccurabout1in5.5millionforfingerprintanalysiscomparison;1in1millionforbitemarkcomparison;1in1millionforhaircomparison;and1in100thousandforhandwritingcomparison.94Whilepreciseerrorratesarenotknownformostofthesetechniques,allindicationspointtotheactualerrorratesbeingordersofmagnitudehigher.Forexample,theFBI’sownstudiesoflatentfingerprintanalysispointtoerrorratesintherangeofoneinseveralhundred.95(Becausetheterm“match”islikelytoimplyaninappropriatelyhighprobativevalue,amoreneutraltermshouldbeusedforanexaminer’sbeliefthattwosamplescomefromthesamesource.Wesuggesttheterm“proposedidentification”toappropriatelyconveytheexaminer’sconclusion,alongwiththepossibilitythatitmightbewrong.Wewillusethistermthroughoutthisreport.)

ThischapterlaysoutPCAST’sconclusionsconcerningthescientificcriteriaforscientificvalidity.Theconclusionsarebasedonthefundamentalprinciplesofthe“scientificmethod”—applicablethroughoutscience—thatvalid

91SeeExACdatabase:exac.broadinstitute.org/gene/ENSG00000197102.92https://www.justice.gov/olp/file/861906/download.93U.S.v.Baines573F.3d979(2009)at984.94JonathanJ.Koehler.“Intuitiveerrorrateestimatesfortheforensicsciences.”(August2,2016).AvailableatSSRN:http://ssrn.com/abstract=2817443orhttp://dx.doi.org/10.2139/ssrn.2817443.95SeeSection5.4.


27

scientificknowledgecanonlybegainedthroughempiricaltestingofspecificpropositions.96Theconclusionsofthechaptermightbebrieflysummarizedasfollows:

Scientificvalidityandreliabilityrequirethatamethodhasbeensubjectedtoempiricaltesting,underconditionsappropriatetoitsintendeduse,thatprovidesvalidestimatesofhowoftenthemethodreachesanincorrectconclusion.Forsubjectivefeature-comparisonmethods,appropriatelydesignedblack-boxstudiesarerequired,inwhichmanyexaminersrenderdecisionsaboutmanyindependenttests(typically,involving“questioned”samplesandoneormore“known”samples)andtheerrorratesaredetermined.Withoutappropriateestimatesofaccuracy,anexaminer’sstatementthattwosamplesaresimilar—orevenindistinguishable—isscientificallymeaningless:ithasnoprobativevalue,andconsiderablepotentialforprejudicialimpact.Nothing—nottraining,personalexperiencenorprofessionalpractices—cansubstituteforadequateempiricaldemonstrationofaccuracy.

Thechapterisorganizedasfollows:

• Thefirstsectiondescribesthedistinctionbetweentwofundamentallydifferenttypesoffeature-comparisonmethods:objectivemethodsandsubjectivemethods.

• Thenextfivesectionsdiscussthescientificcriteriaforthetwotypesofscientificvalidity:foundationalvalidityandvalidityasapplied.

• Thefinaltwosectionsdiscussviewsheldintheforensiccommunity.

4.1 Feature-ComparisonMethods:ObjectiveandSubjectiveMethods

Aforensicfeature-comparisonmethodisaprocedurebywhichanexaminerseekstodeterminewhetheranevidentiarysample(e.g.,fromacrimescene)isorisnotassociatedwithasourcesample(e.g.,fromasuspect)97basedonsimilarfeatures.TheevidentiarysamplemightbeDNA,hair,fingerprints,bitemarks,toolmarks,bullets,tiretracks,voiceprints,visualimages,andsoon.Thesourcesamplewouldbebiologicalmaterialoranitem(tool,gun,shoe,ortire)associatedwiththesuspect.

Feature-comparisonmethodsmaybeclassifiedaseitherobjectiveorsubjective.Byobjectivefeature-comparisonmethods,wemeanmethodsconsistingofproceduresthatareeachdefinedwithenoughstandardizedandquantifiabledetailthattheycanbeperformedbyeitheranautomatedsystemorhumanexaminersexercisinglittleornojudgment.Bysubjectivemethods,wemeanmethodsincludingkeyprocedures

96Forexample,theOxfordOnlineDictionarydefinesthescientificmethodhasbeendefined"amethodorprocedurethathascharacterizedthenaturalsciencessincethe17thcentury,consistinginsystematicobservation,measurement,andexperimentation,andtheformulation,testing,andmodificationofhypotheses."“Scientificmethod”OxfordDictionariesOnline.OxfordUniversityPress,n.d.Web.19Aug2016.97A“sourcesample”referstoaspecificindividualorobject(e.g.,atireorgun).


28

thatinvolvesignificanthumanjudgment—forexample,aboutwhichfeaturestoselectorhowtodeterminewhetherthefeaturesaresufficientlysimilartobecalledaproposedidentification.

Objectivemethodsare,ingeneral,preferabletosubjectivemethods.Analysesthatdependonhumanjudgment(ratherthanaquantitativemeasureofsimilarity)areobviouslymoresusceptibletohumanerror,bias,andperformancevariabilityacrossexaminers.98Incontrast,objective,quantifiedmethodstendtoyieldgreateraccuracy,repeatabilityandreliability,includingreducingvariationinresultsamongexaminers.Subjectivemethodscanevolveintoorbereplacedbyobjectivemethods.99

4.2FoundationalValidity:RequirementforEmpiricalStudies

Forametrologicalmethodtobescientificallyvalidandreliable,theproceduresthatcompriseitmustbeshown,basedonempiricalstudies,toberepeatable,reproducible,andaccurate,atlevelsthathavebeenmeasuredandareappropriatetotheintendedapplication.100,101

BOX2.Definitionofkeyterms

By“repeatable,”wemeanthat,withknownprobability,anexaminerobtainsthesameresult,whenanalyzingsamplesfromthesamesources.

By“reproducible,”wemeanthat,withknownprobability,differentexaminersobtainthesameresult,whenanalyzingthesamesamples.

By“accurate,”wemeanthat,withknownprobabilities,anexaminerobtainscorrectresultsboth(1)forsamplesfromthesamesource(truepositives),and(2)forsamplesfromdifferentsources(truenegatives).

By“reliability,”wemeanthatamethodhasbeenshown,basedonempiricalstudies,toberepeatable,reproducible,andaccurate.102

By“scientificvalidity,”wemeanthatamethodhasshown,basedonempiricalstudies,tobereliablewithlevelsofrepeatability,reproducibility,andaccuracythatareappropriatetotheintendedapplication.

98Dror,I.E.“Ahierarchyofexpertperformance.JournalofAppliedResearchinMemoryandCognition.”JournalofAppliedResearchinMemoryandCognition,Vol.5(2016):121-127.99Forexample,beforethedevelopmentofobjectivetestsforintoxication,courtshadtorelyexclusivelyonthetestimonyofpoliceofficersandotherswhointurnreliedonbehavioralindicationsofdrunkennessandthepresenceofalcoholonthebreath.Thedevelopmentofobjectivechemicaltestsdroveachangefromsubjectivetoobjectivestandards.100NationalPhysicalLaboratory.(2010)“ABeginner’sGuidetoMeasurement”.Availableat:http://www.npl.co.uk/upload/pdf/NPL-Beginners-Guide-to-Measurement.pdf;Pavese,F.“AnIntroductiontoDataModellingPrinciplesinMetrologyandTesting.”InDataModelingforMetrologyandTestinginMeasurementScience,Pavese,F.andForbes,A.B.Eds.(2009).Birkhäuser.101Feature-comparisonmethodsthatgetthewronganswertoooftenhave,bydefinition,lowprobativevalue.Asdiscussedabove,theprejudicialimpactwillthuslikelytooutweightheprobativevalue.102Wenotethat“reliability”alsohasanarrowmeaningwithinthefieldofstatisticsreferringto“consistency”—thatis,theextenttowhichamethodproducesthesameresult,regardlessofwhethertheresultisaccurate.Thisisnotthesenseinwhich“reliability”isusedinthisreport,orinthelaw.


29

Byan“empiricalstudy”,wemeantestinwhichamethodhasbeenusedtoanalyzealargenumberofindependentsetsofsamples,similarinrelevantaspectstothoseencounteredincasework,inordertoestimatethemethod’srepeatability,reproducibility,andaccuracy.

Bya“black-boxstudy”,wemeananempiricalstudythatassessesasubjectivemethodbyhavingexaminersanalyzesamplesandrenderopinionsabouttheoriginorsimilarityofsamples.

Themethodneednotbeperfect,butitisclearlyessentialthatitsaccuracyhasbeenmeasuredbasedonappropriateempiricaltestingandishighenoughtobeappropriatetotheapplication.Withoutanappropriateestimateofitsaccuracy,ametrologicalmethodisuseless—becauseonehasnoideahowtointerpretitsresults.Theimportanceofknowingamethod’saccuracywasemphasizedbythe2009NRCreportonforensicscienceandbya2010NRCreportonbiometrictechnologies.103

Tomeetthescientificcriteriaoffoundationalvalidity,twokeyelementsarerequired:

(1)areproducibleandconsistentprocedurefor(a)identifyingfeatureswithinevidencesamples;(b)comparingthefeaturesintwosamples;and(c)determining,basedonthesimilaritybetweenthefeaturesintwosamples,whetherthesamplesshouldbedeclaredtobeaproposedidentification(“matchingrule”).

(2)empiricalmeasurements,frommultipleindependentstudies,of(a)themethod’sfalsepositiverate—thatis,theprobabilityitdeclaresaproposedidentificationbetweensamplesthatactuallycomefromdifferentsourcesand(b)themethod’ssensitivity—thatis,probabilitythatitdeclaresaproposedidentificationbetweensamplesthatactuallycomefromthesamesource.

Wediscusstheseelementsinturn.

ReproducibleandConsistentProcedures

Foramethodtobeobjective,eachofthethreesteps(featureidentification,featurecomparison,andmatchingrule)shouldbepreciselydefined,reproducibleandconsistent.Forensicexaminersshouldidentifyrelevantfeaturesinthesamewayandobtainthesameresult.Theyshouldcomparefeaturesinthesamequantitativemanner.Todeclareaproposedidentification,theyshouldcalculatewhetherthefeaturesinanevidentiarysampleandthefeaturesinasamplefromasuspectedsourceliewithinapre-specifiedmeasurementtolerance(matchingrule).104Foranobjectivemethod,onecanestablishthefoundationalvalidityofeachoftheindividualstepsbymeasuringitsaccuracy,reproducibility,andconsistency.

103“Biometricrecognitionisaninherentlyprobabilisticendeavor…Consequently,evenwhenthetechnologyandthesystemitisembeddedinarebehavingasdesigned,thereisinevitableuncertaintyandriskoferror.”NationalResearchCouncil,“BiometricRecognition:Challengesandopportunities.”TheNationalAcademiesPress.WashingtonDC.(2010).104Ifasourceisdeclarednottosharethesamefeatures,itis“excluded”bythetest.Thematchingruleshouldbechosencarefully.Ifthe“matchingrule”ischosentobetoostrict,samplesthatactuallycomefromthesamesourcewillbedeclared


30

Forsubjectivemethods,proceduresmuststillbecarefullydefined—buttheyinvolvesubstantialhumanjudgment.Forexample,differentexaminersmayrecognizeorfocusondifferentfeatures,mayattachdifferentimportancetothesamefeatures,andmayhavedifferentcriteriafordeclaringproposedidentifications.Becausetheproceduresforfeatureidentification,thematchingrule,andfrequencydeterminationsaboutfeaturesarenotobjectivelyspecified,theoverallproceduremustbetreatedasakindof“blackbox”insidetheexaminer’shead.

Subjectivemethodsrequirecarefulscrutiny,moregenerally,theirheavyrelianceonhumanjudgmentmeansthattheyareespeciallyvulnerabletohumanerror,inconsistencyacrossexaminers,andcognitivebias.Intheforensicfeature-comparisondisciplines,cognitivebiasincludesthephenomenathat,incertainsettings,humans(i)maytendnaturallytofocusonsimilaritiesbetweensamplesanddiscountdifferences;and(ii)mayalsobeinfluencedbyextraneousinformationandexternalpressuresaboutacase.105(ThelatterissuesareillustratedbytheFBI’smisidentificationofalatentfingerprintintheMadridtrainingbombing,discussedonp.9.)

Sincetheblackboxintheexaminer’sheadcannotbeexamineddirectlyforitsfoundationalbasisinscience,thefoundationalvalidityofsubjectivemethodscanbeestablishedonlythroughempiricalstudiesofexaminer’sperformancetodeterminewhethertheycanprovideaccurateanswers;suchstudiesarereferredtoas“black-box”studies(Box2).Inblack-boxstudies,manyexaminersarepresentedwithmanyindependentcomparisonproblems—typically,involving“questioned”samplesandoneormore“known”samples—andaskedtodeclarewhetherthequestionedsamplescamefromthesamesourceasoneoftheknownsamples.106Theresearchersthendeterminehowoftenexaminersreacherroneousconclusions.

Asanexcellentexample,theFBIrecentlyconductedablack-boxstudyoflatentfingerprintanalysis,involving169examinersand744fingerprintpairs,andpublishedtheresultsofthestudyinaleadingscientificjournal.107

(Someforensicscientistshavecautionedthattoomuchattentiontothesubjectiveaspectsofforensicmethods—suchasstudiesofcognitivebiasandblack-boxstudies—mightdistractfromthegoalofimproving

anon-match(falsenegative).Ifitistoolax,thenthemethodwillnothavemuchdiscriminatorypowerbecausetherandommatchprobabilitywillbetoohigh(falsepositive).105Seee.g.:Boroditsky,L.(2007).Comparisonandthedevelopmentofknowledge.Cognition,102,118-128.;Hassin,R.(2001).Makingfeaturessimilar:comparisonprocessesaffectperception.PsychonomicBulletin&Review,8,728–731.;Medin,D.L.,Goldstone,R.L.,&Gentner,D.(1993).Respectsforsimilarity.PsychologicalReview,100,254–278.;Tversky,A.(1977).Featuresofsimilarity.PsychologicalReview,84,327–352.;Kim,J.,Novemsky,N.,andDhar,R.(2012).Addingsmalldifferencescanincreasesimilarityandchoice.PsychologicalScience,24,225–229.;Larkey,L.B.,&Markman,A.B.(2005).Processesofsimilarityjudgment.CognitiveScience,29,1061–1076.;Medin,D.L.,Goldstone,R.L.,&Markman,A.B.(1995).Comparisonandchoice:Relationsbetweensimilarityprocessesanddecisionprocesses.PsychonomicBulletinandReview,2,1–19.;Goldstone,R.L.(1994).Theroleofsimilarityincategorization:Providingagroundwork.Cognition,52,125–157.;Nosofsky,R.M.(1986).Attention,similarity,andtheidentification-categorizationrelation.JournalofExperimentalPsychology,General,115,39–57.106Answersmaybeexpressedinsuchtermsas“match/nomatch/inconclusive”or“identification/exclusion/inconclusive.”107Ulery,B.T.,Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Accuracyandreliabilityofforensiclatentfingerprintdecisions.”ProceedingsoftheNationalAcademyofSciences,Vol.108,No.19(2011):7733-8.


31

knowledgeabouttheobjectivefeaturesoftheforensicevidenceanddevelopingtrulyobjectivemethods.108Othershavenotedthatthisisnotcurrentlyaproblem,becausecurrenteffortsandfundingtoaddressthechallengesassociatedwithsubjectiveforensicmethodsareverylimited.109)

EmpiricalMeasurementsofAccuracy

Itisnecessarytohaveappropriateempiricalmeasurementsofamethod’sfalsepositiverateandthemethod’ssensitivity.AsexplainedinAppendixA,itisnecessarytoknowthesetwomeasurestoassesstheprobativevalueofamethod.

Thefalsepositiverateistheprobabilitythatthemethoddeclaresaproposedidentificationbetweensamplesthatactuallycomefromdifferentsources.Forexample,afalsepositiverateof5%meansthattwosamplesfromdifferentsourceswill(duetolimitationsofthemethod)beincorrectlydeclaredtocomefromthesamesource5%ofthetime.

Themethod’ssensitivityistheprobabilitythatthemethoddeclaresaproposedidentificationbetweensamplesthatactuallycomefromthesamesource.Forexample,asensitivityof90%meanstwosamplesfromthesamesourcewillbedeclaredtocomefromthesamesource90%ofthetime,anddeclaredtocomefromdifferentsources10%ofthetime.(Thelatterquantityisreferredtoasthefalsenegativerate.)

Thefalsepositiverateisespeciallyimportantbecausefalsepositiveresultscanleadtowrongfulconvictions.110Insomecircumstances,itmaybepossibletoestimateafalsepositiveraterelatedtospecificfeaturesoftheevidenceinthecase.(Forexample,therandommatchprobabilitycalculatedinDNAanalysisdependsinpartonthespecificgenotypeseeninanevidentiarysample.Thefalsepositiverateforlatentfingerprintanalysismaydependonthequalityofthelatentprint.)Forotherfeature-comparisonmethods,itmaybeonlypossibletomakeanoverallestimateoftheaveragefalsepositiverateacrosssamples.

Forobjectivemethods,thefalsepositiverateiscomposedoftwodistinguishablesources—coincidentalmatches(wheresamplesfromdifferentsourcesnonethelesshavefeaturesthatfallwithinthetoleranceoftheobjectivematchingrule)andhuman/technicalfailures(wheresampleshavefeaturesthatfalloutsidethematchingrule,butwhereaproposedidentificationwasnonethelessdeclaredduetoahumanortechnicalfailure).Forobjectivemethodswheretheprobabilityofcoincidentalmatchisverylow(suchasDNAanalysis),thefalsepositiverateinapplicationinagivencasewillbedominatedbytherateofhuman/technicalfailures—whichmaywellbehundredsoftimeslarger.

Forsubjectivemethods,bothtypesoferror—coincidentalmatchesandhuman/technicalfailures—occuraswell,but,withoutanobjective“matchingrule,”thetwosourcescannotbedistinguished.Inestablishingfoundational

108Champod,C.(2014).Researchfocusedmainlyonbiaswillparalyseforensicscience.Science&Justice,54,107–109.109Risinger,D.M.,Thompson,W.C.,Jamieson,A.,Koppl,R.,Kornfield,I.,Krane,D.,Mnookin,J.L.,Rosenthal,R.,Saks,M.J.,&Zabell,S.L.(2014).RegardingChampod,editorial:“Researchfocusedmainlyonbiaswillparalyseforensicscience”.ScienceandJustice,54(6):508-9.110Seefootnote89,p.25.


32

validity,itisthusessentialtoperformblack-boxstudiesthatempiricallymeasuretheoverallerrorrateacrossmanyexaminers.(SeeBox3concerningtheword“error”.)

BOX3.Themeaningsof“error”

Theterm“error”hasdifferingmeaningsinscienceandlaw,whichcanleadtoconfusion.Inlegalsettings,theterm“error”oftenimpliesfault—e.g.,thatapersonhasmadeamistakethatcouldhavebeenavoidedifheorshehadproperlyfollowedcorrectproceduresoramachinehasgivenanerroneousresultthatcouldhavebeenavoideditifhadbeenproperlycalibrated.Inscience,theterm“error”alsoincludesthesituationinwhichtheprocedureitself,whenproperlyapplied,doesnotyieldthecorrectanswerowingtochanceoccurrence.

Whenoneappliesaforensicfeature-comparisonmethodwiththegoalofassessingwhethertwosamplesdidordidnotcomefromthesamesource,coincidentalmatchesandhuman/technicalfailuresarebothregarded,fromastatisticalpointofview,as“errors”becausebothcanleadtoincorrectconclusions.

Studiesdesignedtoestimateamethod’sfalsepositiverateandsensitivityarenecessarilyconductedusingonlyafinitenumberofsamples.Asaconsequence,theycannotprovide“exact”valuesforthesequantities(andshouldnotclaimtodoso),butonly“confidenceintervals,”whoseboundsreflect,respectively,therangeofvaluesthatarereasonablycompatiblewiththeresults.Whenreportingafalsepositiveratetoajury,itisscientificallyimportanttostatethe“upper95%one-sidedconfidencebound”toreflectthefactthattheactualfalsepositiveratecouldreasonablybeashighasthisvalue.111(Formoreinformation,seeAppendixA.)

Studiesoftencategorizetheirresultsasbeingconclusive(e.g.,identificationorexclusion)orinconclusive(nodeterminationmade).112Whenreportingafalsepositiveratetoajury,itisscientificallyimportanttocalculatetheratebasedontheproportionofconclusiveexaminations,ratherthanjusttheproportionofallexaminations.Thisisappropriatebecauseevidenceusedagainstadefendantwilltypicallybebasedonconclusive,ratherthaninconclusive,examinations.Toillustratethepoint,consideranextremecaseinwhichamethodhadbeentested1000timesandfoundtoyield990inconclusiveresults,10falsepositives,andnocorrectresults.Itwouldbemisleadingtoreportthatthefalsepositiveratewas1%(10/1000examinations).Rather,oneshouldreportthat100%oftheconclusiveresultswerefalsepositives(10/10examinations).

111Theupperconfidenceboundproperlyincorporatestheprecisionoftheestimatebasedonthesamplesize.Forexample,ifastudyfoundnoerrorsin100tests,itwouldbemisleadingtotellajurythattheerrorratewas0%.Infact,ifthetestsareindependent,theupper95%confidenceboundforthetrueerrorrateis3.0%.Accordinglyajuryshouldbetoldthattheerrorratecouldbeashighas3.0%(thatis,1in33).Thetrueerrorratecouldbehigher,butwithrathersmallprobability(<5%).Ifthestudyweremuchsmaller,theupper95%confidencelimitwouldbehigher.Forastudythatfoundnoerrorsin10tests,theupper95%confidenceboundis26%--thatis,theactualfalsepositiveratecouldberoughly1in4(seeAppendixA).112SeeChapter5.


33

Whereasexploratoryscientificstudiesmaytakemanyforms,scientificvalidationstudies—intendedtoassessthevalidityandreliabilityofametrologicalmethodforaparticularforensicfeature-comparisonapplication—mustsatisfyanumberofcriteria,whicharedescribedinBox4.

BOX4.Keycriteriaforvalidationstudiestoestablishfoundationalvalidity

Scientificvalidationstudies—intendedtoassessthevalidityandreliabilityofametrologicalmethodforaparticularforensicfeature-comparisonapplication—mustsatisfyanumberofcriteria.(1)Thestudiesmustinvolveasufficientlylargenumberofexaminersandmustbebasedonsufficientlylargecollectionsofknownandrepresentativesamplesfromrelevantpopulationstoreflecttherangeoffeaturesorcombinationsoffeaturesthatwilloccurintheapplication.Inparticular,thesamplecollectionsshouldbe:

(a)representativeofthequalityofevidentiarysamplesseeninrealcases.(Forexample,ifamethodistobeusedondistorted,partial,latentfingerprints,onemustdeterminetherandommatchprobability—thatis,theprobabilitythatthematchoccurredbychance—fordistorted,partial,latentfingerprints;therandommatchprobabilityforfullscannedfingerprints,orevenveryhighqualitylatentprintswouldnotberelevant.)

(b)chosenfrompopulationsrelevanttorealcases.Forexample,forfeaturesinbiologicalsamples,thefalsepositiverateshouldbedeterminedfortheoverallUSpopulationandformajorethnicgroups,asisdonewithDNAanalysis).

(c)largeenoughtoprovideappropriateestimatesoftheerrorrates.

(2)Theempiricalstudiesshouldbeconductedsothatneithertheexaminernorthosewithwhomtheexaminerinteractshaveanyinformationaboutthecorrectanswer.

(3)Thestudydesignandanalysisframeworkshouldbespecifiedinadvance.Invalidationstudies,itisinappropriatetomodifytheprotocolafterwardsbasedontheresults.113

(4)Theempiricalstudiesshouldbeconductedoroverseenbyindividualsororganizationsthathavenostakeintheoutcomeofthestudies.114

113Theanalogoussituationinmedicineisaclinicaltrialtotestthesafetyandefficacyofadrugforaparticularapplication.Inthedesignofclinicaltrials,FDArequiresthatcriteriaforanalysismustbepre-specifiedandnotesthatposthocchangestotheanalysiscompromisethevalidityofthestudy.See:FDAGuidance:“AdaptiveDesignsforMedicalDeviceClinicalStudies”(2016)Availableat:http://www.fda.gov/downloads/medicaldevices/deviceregulationandguidance/guidancedocuments/ucm446729.pdf;Aloshetal.,Statisticalconsiderationsonsubgroupanalysisinclinicaltrials.”StatisticsinBiopharmaceuticalResearch,September2015.;FDAGuidance:“DesignConsiderationsforPivotalClinicalInvestigationsforMedicalDevices”(2013)Availableat:http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm373750.htm;FDAGuidanceforIndustry:E9StatisticalPrinciplesforClinicalTrials(September1998)Availableat:http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm073137.pdf;Pocock,SJ.Clinicaltrials:apracticalapproach,Wiley,Chichester(1983).


34

(5)Data,softwareandresultsfromvalidationstudiesshouldbeavailabletoallowotherscientiststoreviewtheconclusions.

(6)Toensurethatconclusionsarereproducibleandrobust,thereshouldbemultiplestudiesbyseparategroupsreachingsimilarconclusions.

Anempiricalmeasurementoferrorratesisnotsimplyadesirablefeature;itisessentialfordeterminingwhetheramethodisfoundationallyvalid.Inscience,atestingprocedure—suchastestingwhetherapersonispregnantorwhetherwateriscontaminated—isnotconsideredvaliduntilitsreliabilityhasbeenempiricallymeasured.Forexample,weneedtoknowhowoftenthepregnancytestdeclaresapregnancywhenthereisnone,andviceversa.Thesamescientificprinciplesapplynolesstoforensictests,whichmaycontributetoadefendantlosinghislifeorliberty.

Importantly,errorratescannotbeinferredfromcasework,butrathermustbedeterminedbasedonsampleswherethecorrectanswerisknown.Forexample,theformerheadoftheFBI’sfingerprintunittestifiedthattheFBIhad“anerrorrateofoneperevery11millioncases”basedonthefactthattheagencywasknowntohavemadeonlyonemistakeoverthepast11years,duringwhichtimeithadmade11millionidentifications.115Thefallacyisobvious:theexpertsimplyassumedwithoutevidencethateveryerrorincaseworkhadcometolight.

Whyisitessentialtoknowamethod’sfalsepositiverateandsensitivity?Becausewithoutappropriateempiricalmeasurementofamethod’saccuracy,thefactthattwosamplesinaparticularcaseshowsimilarfeatureshasnoprobativevalue—and,asnotedabove,itmayhaveconsiderableprejudicialimpactbecausejurieswilllikelyincorrectlyattachmeaningtotheobservation.116

AdecisionbyU.S.DistrictJudgeJohnPotterinU.S.v.Yee(1991),anearlycaseontheuseofDNAanalysis,elegantlyexpressestheabsoluteneed,fromascientificperspective,forempiricaldata:

Withouttheprobabilityassessment,thejurydoesnotknowwhattomakeofthefactthatthepatternsmatch:thejurydoesnotknowwhetherthepatternsareascommonaspictureswithtwoeyes,orasuniqueastheMonaLisa.117,118

114Inthesettingofclinicaltrials,thesponsorofthetrial(apharmaceutical,deviceorbiotechcompanyor,insomecases,anacademicinstitutions)fundsandinitiatesthestudy,butthetrialisconductedbyindividualswhoareindependentofthesponsor(often,academicphysicians),inordertoensurethereliabilityofthedatageneratedbythestudyandminimizethepotentialforbias.See,e.g.,21CFR§312.3and21CFR§54.4(a).115U.S.v.Baines573F.3d979(2009)at984.116UnderFederalRulesofEvidence,Rule403,evidenceshouldbeexcluded“ifitsprobativevalueissubstantiallyoutweighedbythedangerofunfairprejudice.”117U.S.v.Yee,134F.R.D.161(N.D.Ohio1991).


35

4.3FoundationalValidity:RequirementforScientificallyValidTestimony

Itshouldbeobvious—butitbearsemphasizing—thatonceamethodhasbeenestablishedasfoundationallyvalidbasedonappropriateempiricalstudies,claimsaboutthemethod’saccuracyandtheprobativevalueofproposedidentifications,inordertobevalid,mustbebasedonsuchempiricalstudies.Statementsclaimingorimplyinggreatercertaintythandemonstratedbyempiricalevidencearescientificallyinvalid.Forensicexaminersshouldthereforereportfindingsofaproposedidentificationwithclarityandrestraint,explainingineachcasethatthefactthattwosamplessatisfyamethod’scriteriaforaproposedmatchdoesnotnecessarilyimplythatthesamplescomefromacommonsource.Ifthefalsepositiverateofamethodhasbeenfoundtobe1in50,expertsshouldnotimplythatthemethodisabletoproduceresultsatahigheraccuracy.

Troublingly,expertwitnessessometimesgobeyondtheempiricalevidenceaboutthefrequencyoffeatures—eventotheextentofclaimingorimplyingthatasamplecamefromaspecificsourcewithnear-certaintyorevenabsolutecertainty,despitehavingnoscientificbasisforsuchopinions.119Fromthestandpointofscientificvalidity,expertsshouldneverbepermittedtostateorimplyincourtthattheycandrawconclusionswithcertaintyornear-certainty(suchas‘zero’,‘vanishinglysmall,’‘essentiallyzero,’‘negligible,’‘minimal,’or‘microscopic’errorrates;‘100%certainty’or‘toareasonabledegreeofscientificcertainty’;oridentification‘totheexclusionofallothersources’.120

ThescientificinappropriatenessofsuchtestimonywasaptlysummarizedbyDistrictofColumbiaCourtofAppealsJudgeCatharineEasterlyinWilliamsv.UnitedStates,acaseinwhichanexaminertestifiedthatmarkingsoncertainbulletswereuniquetoagunrecoveredfromadefendant’sapartment:

Asmatterscurrentlystand,acertaintystatementregardingtoolmarkpatternmatchinghasthesameprobativevalueasthevisionofapsychic:itreflectsnothingmorethantheindividual’sfoundationlessfaithinwhathebelievestobetrue.Thisisnotevidenceonwhichwecaningoodconsciencerely,particularlyincriminalcases,wherewedemandproof—realproof—beyondareasonabledoubt,preciselybecausethestakesaresohigh.121

Inscience,assertionsthatametrologicalmethodismoreaccuratethanhasbeenempiricallydemonstratedarerightlyregardedasmerespeculation,notvalidconclusionsthatmeritcredence.

118Somecourtshaveruledthatthereisnoharminadmittingfeature-comparisonevidenceonthegroundsthatjurorscanseethefeatureswiththeirowneyesanddecideforthemselvesaboutwhetherfeaturesareshared.U.S.v.Yeeshowswhythisreasoningisfallacious:jurorshavenowaytoknowhowoftentwodifferentsampleswouldsharefeatures,andtowhatlevelofspecificity.119Asnotedabove,thelonghistoryofexaggeratedclaimsfortheaccuracyofforensicmethodsincludestheDOJ’sownpriorstatementthatlatentfingerprintanalysiswas“infallible”,whichtheDOJhasjudgedtohavebeeninappropriate.119https://www.justice.gov/olp/file/861906/download.120Cole,S.A.“Grandfatheringevidence:FingerprintadmissibilityrulingsfromJenningstoLleraPlazaandbackagain.”41AmericanCriminalLawReview,1189(2004).Seealso:NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(NRCReport,2009)pp.87-;104-;143.121Williamsv.UnitedStates,DCCourtofAppeals,decidedJanuary21,2016,(Easterly,concurring).


36

4.4NeitherExperiencenorProfessionalPracticesCanSubstituteforFoundationalValidity

Insomesettings,anexpertmaybescientificallycapableofrenderingjudgmentsbasedprimarilyonhisorher“experience”and“judgment.”Basedonexperience,asurgeonmightbescientificallyqualifiedtoofferajudgmentaboutwhetheranotherdoctoractedappropriatelyintheoperatingtheaterorapsychiatristmightbescientificallyqualifiedtoofferajudgmentaboutwhetheradefendantwasmentallycompetentatthetimeacrimewascommitted.

Bycontrast,“experience”or“judgment”cannotbeusedtoestablishthescientificvalidityandreliabilityofametrologicalmethod,suchasaforensicfeature-comparisonmethod.Thefrequencywithwhichaparticularpatternorsetoffeatureswillbeobservedindifferentsamples,whichisanessentialelementindrawingconclusions,isnotamatterof“judgment.”Itisanempiricalmatterforwhichonlyempiricalevidenceisrelevant.Moreover,aforensicexaminer’s“experience”fromextensivecaseworkisnotinformative—becausethe“rightanswers”arenottypicallyknownincaseworkandthusexaminerscannotaccuratelyknowhowoftentheyerroneouslydeclarematchesandcannotreadilyhonetheiraccuracybylearningfromtheirmistakesinthecourseofcasework.

Importantly,goodprofessionalpractices—suchastheexistenceofprofessionalsocieties,certificationprograms,accreditationprograms,peer-reviewedarticles,standardizedprotocols,proficiencytesting,andcodesofethics—cannotsubstituteforactualevidenceofscientificvalidityandreliability.122

Similarly,anexpert’sexpressionofconfidencebasedonpersonalprofessionalexperienceorexpressionsofconsensusamongpractitionersabouttheaccuracyoftheirfieldisnosubstituteforerrorratesestimatedfromrelevantstudies.Foramethodtobereliable,empiricalevidenceofvalidity,asdescribedabove,isrequired.

Finally,thepointsaboveunderscorethatscientificvalidityofamethodmustbeassessedwithintheframeworkofthebroaderscientificfieldofwhichitisapart(e.g.,measurementscienceinthecaseoffeature-comparisonmethods).Thefactthatbitemarkexaminersdefendthevalidityofbitemarkexaminationmeanslittle.

4.5ValidityasApplied:KeyElements

Foundationalvaliditymeansthatamethodcan,inprinciple,bereliable.Validityasappliedmeansthatthemethodhasbeenreliablyappliedinpractice.Itisthescientificconceptwemeantocorrespondtothelegalrequirement,inRule702(d),thatanexpert“hasreliablyappliedtheprinciplesandmethodstothefactsofthecase.”

Fromascientificstandpoint,certaincriteriaareessentialtoestablishthataforensicpractitionerhasreliablyappliedamethodtothefactsofacase.TheseelementsaredescribedinBox5.

122Forexample,bothscientificandpseudoscientificdisciplines(includingpsychics)employsuchpractices.


37

BOX5.Keycriteriaforvalidityasapplied

(1)Theforensicexaminermusthavebeenshowntobecapableofreliablyapplyingthemethodandmustactuallyhavedoneso.Demonstratingthatanexamineriscapableofreliablyapplyingthemethodiscrucial—especiallyforsubjectivemethods,inwhichhumanjudgmentplaysacentralrole.Fromascientificstandpoint,theabilitytoapplyamethodreliablycanbedemonstratedonlythroughempiricaltestingthatmeasureshowoftentheexpertreachesthecorrectanswer.(Proficiencytestingisdiscussedmoreextensivelyonp.38.)Determiningwhetheraexaminerhasactuallyreliablyappliedthemethodrequiresthattheproceduresactuallyusedinthecase,theresultsobtained,andthelaboratorynotesbemadeavailableforscientificreviewbyothers.

(2)Assertionsabouttheprobativevalueofproposedidentificationsmustbescientificallyvalid.

(a)Theforensicexaminershouldreporttheoverallfalsepositiverateandsensitivityforthemethodestablishedinthestudiesoffoundationalvalidityandshoulddemonstratethatthesamplesusedinthefoundationalstudiesarerelevanttothefactsofthecase.123

(b)Whereapplicable,theexaminershouldreporttheprobativevalueoftheobservedmatchbasedonthespecificfeaturesobservedinthecase.124

(c)Anexpertshouldnotmakeclaimsorimplicationsthatgobeyondtheempiricalevidenceandtheapplicationsofvalidstatisticalprinciplestothatevidence.

4.6ValidityasApplied:ProficiencyTesting

Evenwhenamethodisfoundationallyvalid,therearemanyreasonswhyexaminersmaynotalwaysgettherightresult.125Asdiscussedabove,theonlywaytoestablishscientificallythatanexamineriscapableofapplying

123Forexample,forDNAanalysis,thefrequencyofgeneticvariantsisknowntovaryamongethnicgroups;itisthusimportantthatthesamplecollectionreflectrelevantethnicgroupstothecaseathand.Forlatentfingerprints,theriskoffalselydeclaringanidentificationmaybehigherwhenlatentfingerprintsareoflowerquality;so,toberelevant,thesamplecollectionsusedtoestimateaccuracyshouldbebasedonlatentfingerprintscomparableinqualityandcompletenesstothecaseathand.124Therelevantquestionis,“Whatistheprobabilitythataspecifictwosamplesthathavebeendeclaredto‘match’actuallycomefromthesamesource?”Statisticiansrefertothisprobabilityasthe“positivepredictivevalue”(PPV)ofthetest.Itdependsonthefalsepositiverate,sensitivity,poolsizeandpossiblyotherfactors(seeAppendixA).125J.J.Koehlerhasenumeratedanumberofpossibleproblemsthatcould,inprinciple,occur:featuresmaybemismeasured;samplesmaybeinterchanged,mislabeled,miscoded,altered,orcontaminated;equipmentmaybemiscalibrated;technicalglitchesandfailuresmayoccurwithoutwarningandwithoutbeingnoticed;andresultsmaybemisread,misinterpreted,misrecorded,mislabeled,mixedup,misplaced,ordiscarded.Koehler,J.J.“Forensicsorfauxrensics?Testingforaccuracyintheforensicsciences.”papers.ssrn.com/sol3/papers.cfm?abstract_id=2773255(accessedJune28,2016).


38

afoundationallyvalidmethodisthroughappropriateempiricaltestingtomeasurehowoftentheexaminergetsthecorrectanswer.

Suchempiricaltestingisoftenreferredtoas“proficiencytesting.”Wenotethatterm“proficiencytesting”issometimesusedtorefertomanydifferentothertypesoftesting—suchas(i)teststodeterminewhetherapractitionerreliablyfollowsthestepslaidoutinaprotocol,withoutassessingtheaccuracyoftheirconclusions,and(ii)practiceexercisesthathelppractitionersimprovetheirskillsbyhighlightingtheirerrors,withoutaccuratelyreflectthecircumstancesofactualcasework.

Inthisreport,weusethetermproficiencytestingtomeanempiricalteststoassessanexaminer’saccuracy.126

Proficiencytestingshouldbeperformedonsamplesforwhichthetrueanswerisknown,underconditionsrepresentativeofcaseworkandusingsamplesrepresentativeoftheintendedapplication.(Forexample,thefactthatanexaminerpassesaproficiencytestinvolvingDNAanalysisofsimple,single-sourcesamplesdoesnotdemonstratethattheyarecapableofDNAanalysisofcomplexmixtures;seep.48-50.)

Toensureintegrity,proficiencytestingshouldbeoverseenbyadisinterestedthirdpartythathasnoinstitutionalorfinancialincentivetoskewperformance.Wenotethattestingserviceshavestatedthatforensiccommunityprefersthattestsnotbetoochallenging..127

Asnotedabove,falsepositiveratesconsistofbothcoincidentalmatchratesandtechnical/humanfailurerates.Forsometechnologies(suchasDNAanalysis),thelattermaybehundredsoftimeshigherthantheformer.

Proficiencytestingisespeciallycriticalforsubjectivemethods:becausetheprocedureisnotbasedsolelyonobjectivecriteriabutreliesonhumanjudgment,itisinherentlyvulnerabletoerrorandinter-examinervariability.Eachexaminershouldbetested,becauseempiricalstudieshavenotedconsiderabledifferencesinaccuracyacrossexaminers.128,129

Thetestproblemsusedinproficiencytestsshouldbepubliclyreleasedafterthetestiscompleted,toenablescientiststoassesstheappropriatenessandadequacyofthetestfortheirintendedpurpose.

126Wenotethatproficiencytestingisnoteintendedtoestimatetheinherenterrorratesofamethod;theseratesshouldbeassessedfromfoundationalvaliditystudies.127ChrisCzyryca,thepresidentofCollaborativeTestingServices,Inc.,theleadingproficiencytestingfirmintheU.S.,haspubliclystatedthat“Easytestsarefavoredbythecommunity.”August2015meetingoftheNationalCommissiononForensicScience,apresentationattheAccreditationandProficiencyTestingSubcommittee.www.justice.gov/ncfs/file/761061/download.128Forexample,a2011studyonlatentfingerprintdecisionsobservedthatexaminersfrequentlydifferedonwhetherfingerprintsweresuitableforreachingaconclusion.Ulery,B.T.,Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Accuracyandreliabilityofforensiclatentfingerprintdecisions.”ProceedingsoftheNationalAcademyofSciences,Vol.108,No.19(2011):7733-8.129Itisnotsufficienttopointtoproficiencytestingonvolunteersinalaboratory,becausebetterperformingexaminersaremorelikelytoparticipate.Koehler,J.J.“Forensicsorfauxrensics?Testingforaccuracyintheforensicsciences.”papers.ssrn.com/sol3/papers.cfm?abstract_id=2773255(accessedJune28,2016).


39

Finally,proficiencytestingshouldideallybeconductedina‘test-blind’manner—thatis,withsamplesinsertedintotheflowofcaseworksuchthatexaminersdonotknowthattheyarebeingtested.(Forexample,theTransportationSecurityAdministrationconductsblindtestsbysendingweaponsandexplosivesinsideluggagethroughscreeningcheckpointstoseehowoftenTSAscreenersdetectthem.)Ithasbeenestablishedinmanyfields(includinglatentfingerprintanalysis)that,whenindividualsareawarethattheyarebeingtested,theyperformdifferentlythantheydointhecourseoftheirdailywork(referredtoasthe“HawthorneEffect”).130,131

Whiletest-blindproficiencytestingisideal,thereisdisagreementintheforensiccommunityaboutitsfeasibilityinallsettings.Ontheonehand,laboratoriesvaryconsiderablyastothetypeofcasestheyreceive,howevidenceismanagedandprocessed,andwhatinformationisprovidedtoananalystabouttheevidenceorthecaseinquestion.Accordingly,blinded,inter-laboratoryproficiencytestsmaybedifficulttodesignandorchestrateonalargescale.132Ontheotherhand,test-blindproficiencytestshavebeenusedforDNAanalysis,133andselectlabshavebeguntoimplementthistypeoftesting,in-house,aspartoftheirqualityassuranceprograms.134

PCASTbelievesthattest-blindproficiencytestingofforensicexaminersshouldbevigorouslypursued,withtheexpectationthatitshouldbeinwideuse,atleastinlargelaboratories,withinthenextfiveyears.However,PCASTbelievesthatitisnotyetrealistictorequiretest-blindproficiencytestingbecausetheproceduresfortest-blindproficiencytestshavenotyetbeendesignedandevaluated.

130ConcerningtheHawthorneeffect,see,e.g.:Bracht,G.H.,andG.V.Glass.“Theexternalvalidityofexperiments.”AmericanEducationalResearchJournal,Vol.5,No.4(1968):437-74;Weech,T.L.andH.Goldhor."ObtrusiveversusunobtrusiveevaluationofreferenceserviceinfiveIllinoispubliclibraries:Apilotstudy."LibraryQuarterly:Information,Community,Policy,Vol.52,No.4(1982):305-24;Bouchet,C.,Guillemin,F.,andS.Braincon.“Nonspecificeffectsinlongitudinalstudies:impactonqualityoflifemeasures.”JournalofClinicalEpidemiology,Vol.49,No.1(1996):15-20;Mangione-Smith,R.,Elliott,M.N.,McDonald,L.,andE.A.McGlynn.“AnobservationalstudyofantibioticprescribingbehaviorandtheHawthorneEffect.”HealthServicesResearch,Vol.37,No.6(2002):1603-23;Mujis,D.“Measuringteachereffectiveness:Somemethodologicalreflections.”EducationalResearchandEvaluation,Vol.12,No.1(2006):53–74;andMcCarney,R.,Warner,J.,Iliffe,S.,vanHaselen,R.,Griffin,M.,andP.Fisher.“TheHawthorneEffect:arandomized,controlledtrial.”BMCMedicalResearchMethodology,Vol.7,No.30(2007).131Fordemonstrationsthatforensicexaminerschangetheirbehaviorwhentheyknowtheirperformanceisbeingmonitoredinparticularways,seeLangenburg,G.“AperformancestudyoftheACE-Vprocess:Apilotstudytomeasuretheaccuracy,precision,reproducibility,repeatability,andbiasabilityofconclusionsresultingfromtheACE-Vprocess.”JournalofForensicIdentification,Vol.59,No.2(2009).132Someofthechallengesassociatedwithdesigningblindinter-laboratoryproficiencytestsmaybeaddressediftheforensiclaboratoriesweretomovetowardasystemwhereanexaminer’sknowledgeofacasewerelimitedtodomain-relevantinformation.133See:Peterson,J.L.,Lin,G.,Ho,M.,Chen,Y.,andR.E.Gaensslen.“ThefeasibilityofexternalblindDNAproficiencytesting.II.Experiencewithactualblindtests.”JournalofForensicScience,Vol.48,No.1(2003):32-40.134Forexample,theHoustonForensicScienceCenterhasimplementedroutine,blindproficiencytestingforitsfirearmsexaminersandchemistryanalysisunit,andisplanningtocarryoutsimilartestingforitsDNAandlatentprintexaminers.DiscussionwithWilliamC.Thompson,May2,2016.


40

Whileonlynon-test-blindproficiencytestsareusedtosupportvalidityasapplied,itisscientificallyimportanttoreportthislimitation,includingtojuries—because,asnotedabove,non-blindproficiencytestsarelikelytooverestimatetheaccuracybecausetheexaminersknewtheywerebeingtested.

4.7Non-EmpiricalViewsintheForensicCommunity

Whilethescientificvalidityofmetrologicalmethodsrequiresempiricaldemonstrationofaccuracy,therehavehistoricallybeeneffortsintheforensiccommunitytojustifynon-empiricalapproaches.Thisisofparticularconcernbecausesuchviewsaresometimesmistakenlycodifiedinpoliciesorpractices.Theseheterodoxviewstypicallyinvolvefourrecurrentthemes,whichwereviewbelow.

“Theories”ofIdentification

Acommonargumentisthatforensicpracticesshouldberegardedasvalidbecausetheyrestonscientific“theories”akintothefundamentallawsofphysics,thatshouldbeacceptedbecausetheyhavebeentestedandnot“falsified.”135

Anexampleisthe“TheoryofIdentificationasitRelatestoToolmarks,”issuedin2011bytheAssociationofFirearmandToolMarkExaminers.136,137Itstatesinitsentirety:

1.Thetheoryofidentificationasitpertainstothecomparisonoftoolmarksenablesopinionsofcommonorigintobemadewhentheuniquesurfaceoftwotoolmarksarein“sufficientagreement.”

2.This“sufficientagreement”isrelatedtothesignificantduplicationofrandomtoolmarksasevidencedbythecorrespondenceofapatternorcombinationofpatternsofsurfacecontours.Significanceisdeterminedbythecomparativeexaminationoftwoormoresetsofsurfacecontourpatternscomprisedofindividualpeaks,ridgesandfurrows.Specifically,therelativeheightordepth,width,curvatureandspatialrelationshipoftheindividualpeaks,ridgesandfurrowswithinonesetofsurfacecontoursaredefinedandcomparetothecorrespondingfeaturesinthesecondsetofsurfacecontours.Agreementissignificantwhentheagreementinindividualcharacteristicsexceedsthebestagreementdemonstratedbetweentoolmarksknowntohavebeenproducedbydifferenttoolsandisconsistentwithagreementdemonstratedbytoolmarksknowntohavebeenproducedbythesametool.Thestatementthat“sufficientagreement”existsbetweentwotoolmarksmeansthattheagreementofindividualcharacteristicsisofaquantityandqualitythatthelikelihoodanothertoolcouldhavemadethemarkissoremoteastobeconsideredapracticalimpossibility.

3.Currentlytheinterpretationofindividualization/identificationissubjectiveinnature,foundedonscientificprinciplesandbasedontheexaminer’strainingandexperience.

135See:www.swggun.org/index.php?option=com_content&view=article&id=66:the-foundations-of-firearm-and-toolmark-identification&catid=13:other&Itemid=43.136AssociationofFirearmandToolMarkExaminers.“TheoryofIdentificationasitRelatestoToolMarks:Revised.”AFTEJournal,Vol.43,No.4(2011):287.137FirearmsanalysisisconsideredindetailinChapter5.


41

Thestatementisclearlynotascientifictheory,whichtheNationalAcademyofScienceshasdefinedas“acomprehensiveexplanationofsomeaspectofnaturethatissupportedbyavastbodyofevidence”.138Rather,itisaclaimthatexaminersapplyingasubjectiveapproachcanaccuratelyindividualizetheoriginofatoolmark.Moreover,a“theory”isnotwhatisneeded.Whatisneededareempiricalteststoseehowwellthemethodperforms.

Moreimportantly,thestatedmethodiscircular.Itdeclaresthatanexaminermaystatethattwotoolmarkshavea“commonorigin”whentheirfeaturesarein“sufficientagreement.”Itthendefines“sufficientagreement”asoccurringwhentheexaminerconsidersita“practicalimpossibility”thatthetoolmarkshavedifferentorigins.(InresponsetoPCAST’sconcernaboutthiscircularity,theFBILaboratoryrepliedthat:“‘Practicalimpossibility’isthecertitudethatexistswhenthereissufficientagreementinthequalityandquantityofindividualcharacteristics.”139Thisanswerdidnotresolvethecircularity.)

Focuson‘TrainingandExperience’RatherThanEmpiricalDemonstrationofAccuracy

Manypractitionersholdanhonestbeliefthattheyareabletomakeaccuratejudgmentsaboutidentificationbasedontheirtrainingandexperience.ThisnotionisexplicitintheAFTE’sTheoryofIdentificationabove,whichnotesthatinterpretationissubjectiveinnature,“basedonanexaminer’strainingandexperience.”Similarly,theleadingtextbookonfootwearanalysisstates,

Positiveidentificationsmaybemadewithasfewasonerandomidentifyingcharacteristic,butonlyifthatcharacteristicisconfirmable;hassufficientdefinition,clarity,andfeatures;isinthesamelocationandorientationontheshoeoutsole;andintheopinionofanexperiencedexaminer,wouldnotoccuragainonanothershoe.140[emphasisadded]

Ineffect,itsays,positiveidentificationdependsontheexaminerbeingpositiveabouttheidentification.

“Experience”isaninadequatefoundationfordrawingjudgmentsaboutwhethertwosetsoffeaturescouldhavebeenproducedby(orfoundon)differentsources.Evenifexaminerscouldrecallinsufficientdetailallthepatternsorsetsoffeaturesthattheyhaveseen,theywouldhavenowayofknowingaccuratelyinwhichcasestwopatternsactuallycamefromdifferentsources,becausethecorrectanswersarerarelyknownincasework.

Thefallacyofrelyingon“experience”wasevidentintestimonybyaformerheadoftheFBI’sfingerprintunit(discussedabove)thattheFBIhad“anerrorrateofoneperevery11millioncases,”basedonthefactthattheagencywasonlyawareofonemistake.141Bycontrast,recentempiricalstudiesbytheFBILaboratory(discussedinChapter5)indicateerrorratesofroughlyoneinseveralhundred.

138See:www.nas.edu/evolution/TheoryOrFact.html.139CommunicationfromFBILaboratorytoPCAST(June6,2016).140Bodziak,W.J.FootwearImpressionEvidence:Detection,Recovery,andExamination.2nded.CRCPress-Taylor&Francis,BocaRaton,Florida(2000).141U.S.v.Baines573F.3d979(2009)at984.


42

“Training”isanevenweakerfoundation.Themerefactthatanindividualhasbeentrainedinamethoddoesmeanthatthemethoditselfisscientificallyvalidnorthattheindividualiscapableofproducingreliableanswerswhenapplyingthemethod.

Focuson‘Uniqueness’RatherThanAccuracy

Manyforensicfeature-comparisondisciplinesarebasedonthepremisethatvarioussetsoffeatures(forexample,fingerprints,toolmarksonbullets,humandentition,andsoon)are“unique.”142

Theforensicsscienceliteraturecontainsmany“uniqueness”studiesthatgotogreatlengthstotrytoestablishthecorrectnessofthispremise.143Forexample,a2012paper144studied39AdidasSupernovaClassicrunningshoes(size12)wornbyasinglerunnerover8years,duringwhichtimehekeptarunningjournalandranoverthesametypesofsurfaces.Afterapplyingblackshoepolishtothesolesoftheshoes,theauthoraskedtherunnertocarefullyproducetreadmarksonsheetsoflegalpaperonahardwoodfloor.Theauthorshowedthatitwaspossibletoidentifysmallidentifyingdifferencesbetweenthetreadmarksproducedbydifferentpairsofshoes.

142Forfingerprints,see,e.g.,Wertheim,Kasey.“Letterre:ACE-V:Isitscientificallyreliableandaccurate?”JournalofForensicIdentification;Nov/Dec2002;52,6,pg.669(“Thelawofbiologicaluniquenessstatesthatexactreplicationofanygivenorganismcannotoccur(natureneverrepeatsitself),and,therefore,nobiologicalentitywilleverbeexactlythesameasanother”)andBruceBudowle,JoAnnBuscagliaandRebeccaSchwartzPerlman.“Reviewofthescientificbasisforfrictionridgecomparisonsasameansofidentification:committeefindingsandrecommendations.”ForensicScienceCommunications.8.1(Jan.2006).Availableat:https://www2.fbi.gov/hq/lab/fsc/backissu/jan2006/research/2006_01_research02.htm(“Theuseoffrictionridgeskincomparisonsasameansofidentificationisbasedontheassumptionsthatthepatternoffrictionridgeskinisbothuniqueandpermanent”).Forfirearms,see,e.g.,Riva,FabianandChampod,Christope.“AutomaticComparisonandEvaluationofImpressionsLeftbyaFirearmonFiredCartridgeCases.”JournalofForensicSciences,2014,Vol.59(3),p.637.(“Theabilitytoidentifyafirearmasthesourceofaquestionedcartridgecaseorbulletisbasedontwotenetsconstitutingthescientificfoundationofthediscipline[6].Thefirstassumestheuniquenessofimpressionsleftbythefirearms”)andSWGGUNAdmissibilityResourceKit(ARK):FoundationalOverviewofFirearm/ToolmarkIdentification.Availableat:https://afte.org/resources/swggun-ark(“ThebasisforidentificationinToolmarkIdentificationisfoundedontheprincipleofuniqueness...wherein,allobjectsareuniquetothemselvesandthuscanbedifferentiatedfromoneanother”).Forbitemarks,see,e.g.,Kieser,JAetal.“TheUniquenessoftheHumanAnteriorDentition:AGeometricMorphometricAnalysis.”JForensicSci.2007May;52(3):671-7.(“Therearetwopostulatesthatunderlieallbitemarkanalyses:first,thatthecharacteristicsoftheanteriorteethinvolvedinthebiteareunique,andsecondly,thatthisuniquenessisaccuratelyrecordedinthematerialbitten.”)andPretty,IA“ResolvingIssuesinBitemarkAnalysis”inBitemarkEvidence:AColorAtlasRBJDorian,Ed.CRCPress.Chicago2011.(““Bitemarkanalysisisbasedontwopostulates:(a)thedentalcharacteristicsofanteriorteethinvolvedinbitingareuniqueamongindividuals,and(b)thisasserteduniquenessistransferredandrecordedintheinjury.”).143Someauthorshavecriticizedattemptstoaffirmtheuniquenesspropositionbasedonobservations,notingthattheyrestonpureinductivereasoning,amethodforscientificinvestigationthat“felloutoffavourduringtheepochofSirFrancisBaconinthe16thcentury.”Pageetal.(2011)“Uniquenessintheforensicidentificationsciences—Factorfiction?”ForensicScienceInternational,Vol.206(1),pp.13.144Wilson,H.D.“Comparisonoftheindividualcharacteristicsintheoutsolesofthirty-ninepairsofAdidasSupernovaClassicshoes.”JournalofForensicIdentification,Vol.62,No.3(2012):194-204.


43

Yet,uniquenessstudiesmissthefundamentalpoint.Theissueisnotwhetherobjectsorfeaturesdiffer;theysurelydoifonelooksatafineenoughlevel.Theissueishowwellandunderwhatcircumstancesexaminersapplyingagivenmetrologicalmethodcanreliablydetectrelevantdifferencesinfeaturestoreliablyidentifywhethertheyshareacommonsource.Uniquenessstudies,whichfocusonthepropertiesoffeaturesthemselves,canthereforeneverestablishwhetheraparticularmethodformeasuringandcomparingfeaturesisfoundationallyvalid.Onlyempiricalstudiescandoso.

Moreover,itisnotnecessaryforfeaturestobeuniqueinorderforthemtobeusefulinnarrowingdownthesourceofafeature.Rather,itisessentialthattherebeempiricalevidenceabouthowoftenamethodincorrectlyattributesthesourceofafeature.

DecouplingConclusionsaboutIdentificationfromEstimatesofAccuracy

Finally,someholdtheviewthat,whentheapplicationofascientificmethodleadstoaconclusionofanassociationorproposedidentification,itisunnecessarytoreportincourtthereliabilityofthemethod.145Asarationale,itissometimesarguedthatitisimpossibletomeasureerrorratesperfectlyorthatitisimpossibletoknowtheerrorrateinthespecificcaseathand.

Thisnotioniscontrarytothefundamentalprincipleofscientificvalidityinmetrology—namely,thattheclaimthattwoobjectshavebeencomparedandfoundtohavethesameproperty(length,weight,orfingerprintpattern)ismeaninglesswithoutquantitativeinformationaboutthereliabilityofthecomparisonprocess.

Itisstandardpracticetostudyandreporterrorratesinmedicine—bothtoestablishthereliabilityofamethodinprincipleandtoassessitsimplementationinpractice.Noonearguesthatmeasuringorreportingclinicalerrorratesisinappropriatebecausetheymightnotperfectlyreflectthesituationforaspecificpatient.Iftransparencyabouterrorratesisappropriateformatchingbloodtypesbeforeatransfusion,itisappropriateformatchingforensicsamples—whereerrorsmayhavesimilarlife-threateningconsequences.

WereturntothistopicinChapter8,whereweobservethattheDOJ’srecentproposedguidelinesonexperttestimonyarebased,inpart,onthisscientificallyinappropriateview.

4.8EmpiricalViewsintheForensicCommunity

Notwithstandingtheviewsdescribedintheprevioussection,agrowingsegmentoftheforensicsciencecommunityhasrespondedtothe2009NRCreportwithanincreasedrecognitionoftheneedforempiricalstudiesandwithinitialeffortstoundertakethem.Examplesincludepublishedresearchstudiesbyforensicscientists,assessmentsofresearchneedsbySWGandOSACcommittees,andstatementsfromtheNCFS.Belowwehighlightseveralexamplesfromrecentpapersbyforensicscientists:

● ResearchersattheNationalAcademyofSciencesandelsewhere(e.g.,Saks&Koehler,2005;Spinney,2010)havearguedthatthereisanurgentneedtodevelopobjectivemeasuresofaccuracyinfingerprint

145See:www.justice.gov/olp/file/861936/download.


44

identification.Herewepresentsuchdata.146

● Toolmarkimpressionevidence,forexample,hasbeensuccessfullyusedincourtsfordecades,butitsexaminationhaslackedscientific,statisticalproofthatwouldindependentlycorroborateconclusionsbasedonmorphologycharacteristics(2–7).Inourstudy,wewillapplymethodsofstatisticalpatternrecognition(i.e.,machinelearning)totheanalysisoftoolmarkimpressions.147

● TheNASreportcallsforfurtherresearchintheareaofbitemarkstodemonstratethatthereisalevelofprobativevalueandpossiblyrestrictingtheuseofanalysestotheexclusionofindividuals.Thiscalltorespondmustbeheardifbite-markevidenceistobedefensibleaswemoveforwardasadiscipline.148

● TheNationalResearchCounciloftheNationalAcademiesandthelegalandforensicsciencescommunitieshavecalledforresearchtomeasuretheaccuracyandreliabilityoflatentprintexaminers’decisions,achallengingandcomplexprobleminneedofsystematicanalysis.Ourresearchisfocusedonthedevelopmentofempiricalapproachestostudyingthisproblem.149

● Webelievethisreportshouldencouragethelegalcommunitytorequirethattheemergingfieldofforensicneuroimaging,includingfMRIbasedliedetection,haveaproperscientificfoundationbeforebeingadmittedincourts.150

● Anempiricalsolutionwhichtreatsthesystem[referringtovoiceprints]asablackboxanditsoutputaspointvaluesisthereforepreferred.151

Similarly,theOSAChasdevelopeddocumentsthatdetailcriticalresearchgapsintheevidencesupportingvariousforensicsciencedisciplinesasafoundationfordevelopingplanstoclosethesegaps.Wehighlightseveralexamplesbelow:

● Whilevalidationstudiesoffirearmsandtoolmarkanalysisschemeshavebeenconducted,mosthavebeenrelativelysmalldatasets.Ifalargestudywerewelldesignedandhassufficientparticipation,itis

146Tangen,J.M.,Thompson,M.B.,andD.J.McCarthy.“Identifyingfingerprintexpertise.”PsychologicalScience,Vol.22,No.8(2011):995-7.147Petraco,N.etal.(2012)“AddressingtheNationalAcademyofSciences’Challenge:AMethodforStatisticalPatternComparisonofStriatedToolMarks”JournalofForensicSciences,Vol.57(4),pp.900-911.148PrettyIAandSweetD.(2010)“Aparadigmshiftintheanalysisofbitemarks”ForensicScienceInternational.Vol.201(1),pp.38-44.149Uleryetal.(2011)“Accuracyandreliabilityofforensiclatentfingerprintdecisions.”PNAS.vol.108,no.19,7733–7738.150LanglebenandMoriarty(2013)“UsingBrainImagingforLieDetection:WhereScience,Law,andPolicyCollide”Psychology,PublicPolicy,andLaw.Vol.19,No.2,222–234. 151Morrisonetal.(2011)“Anempiricalestimateoftheprecisionoflikelihoodratiosfromaforensic-voice-comparisonsystem”ForensicScienceInternational.Volume208,Pages59–65.


45

ouranticipationthatsimilarlessonscouldbelearnedforthefirearmsandtoolmarkdiscipline.152

● Weareunawareofanystudythatassessestheoverallfirearmandtoolmarkdiscipline’sabilitytocorrectly/consistentlycategorizeevidencebyclasscharacteristics,identifysubclassmarks,andeliminateitemsusingindividualcharacteristics.153

● Currentlythereisnotareliableassessmentofthediscriminatingstrengthofspecificfrictionridgefeaturetypes.154

● Todatethereislittlescientificdatathatquantifiestheoverallriskofclosenon-matchesinAFISdatabases.ItisdifficulttocreatestandardsregardingsufficiencyforexaminationorAFISsearchsearchingwithoutthistypeofresearch.155

● Researchisneededthatstudieswhethersequentialunmaskingreducesthenegativeeffectsofbiasduringlatentprintexamination.156

● TheSWGGUNhasbeenawareofthescientificandsystemicissuesidentifiedin[theNRCreport]reportforsometimeandhasbeenworkingdiligentlytoaddressthem....identifiestheareaswherewemustfundamentallyimproveourprocedurestoenhancethequalityandreliabilityofourscientificresults,aswellasbetterarticulatethebasisofourscience.157

● TheIAIhas,formanyyears,soughtsupportforresearchthatwouldscientificallyvalidatemanyofthecomparativeanalysesconductedbyitsmemberpractitioners.Whilethereisagreatdealofempiricalevidencetosupporttheseexams,independentvalidationhasbeenlacking.158

152OSACResearchNeedsAssessmentForm.“StudytoAssessTheAccuracyandReliabilityofFirearmandToolmark.”IssuedOctober2015(ApprovedJanuary2016).Availableat:http://www.nist.gov/forensics/osac/upload/FATM-Research-Needs-Assessment_Blackbox.pdf153OSACResearchNeedsAssessmentForm.“AssessmentofExaminers’ToolmarkCategorizationAccuracy.”IssuedOctober2015(ApprovedJanuary2016).Availableat:http://www.nist.gov/forensics/osac/upload/FATM-Research-Needs-Assessment_Class-and-individual-marks.pdf154OSACResearchNeedsAssessmentForm.“AssessingtheSufficiencyandStrengthofFrictionRidgeFeatures.”IssuedOctober2015.Availableat:http://www.nist.gov/forensics/osac/upload/FRS-Research-Need-Assessment-of-Features.pdf155OSACResearchNeedsAssessmentForm.“CloseNon-MatchAssessment.”IssuedOctober2015.Availableat:http://www.nist.gov/forensics/osac/upload/FRS-Research-Need-Close-Non-Match-Assessment.pdf 156OSACResearchNeedsAssessmentForm.“ACE-VBias.”IssuedOctober2015.Availableat:http://www.nist.gov/forensics/osac/upload/FRS-Research-Need-ACE-V-Bias.pdf157See:www.swggun.org/index.php?option=com_content&view=article&id=37&Itemid=22158InternationalAssociationforIdentification.LettertoPatrickJ.Leahy,Chairman,SenateCommitteeontheJudiciary,March18,2009.Availableat:https://www.theiai.org/current_affairs/nas_response_leahy_20090318.pdf


46

TheNationalCommissionhasalsorecognizedtheneedforadditionalworkandmadearecommendationtotheAttorneyGeneralconcerningthefundamentalroleofvalidationstudies:

● TheCommissionhasdevelopedaviewsdocumentontheimportanceofdevelopmentalvalidationforforensicsciencetestmethodstoestablishthemethodologicallimitsofthesetestmethodsandtheneedforthesedevelopmentalvalidationstudiestobeperformedanddocumentedpriortothecreationofdocumentarystandardsinvolvingthesesametestmethods.159

PCASTapplaudsthisgrowingfocusonempiricalevidence.Wenotethatincreasedresearchfundingwillbeneededtoachievethesecriticalgoals(seeChapter6).

4.9SummaryofScientificFindings

Wesummarizeourscientificfindingsconcerningthescientificcriteriaforfoundationalvalidityandvalidityasapplied.Finding1:ScientificCriteriaforScientificValidityofaForensicFeature-ComparisonMethod

(1)Foundationalvalidity.Toestablishfoundationalvalidityforaforensicfeature-comparisonmethod,thefollowingelementsarerequired:

(a)areproducibleandconsistentprocedurefor(i)identifyingfeaturesinevidencesamples;(ii)comparingthefeaturesintwosamples;and(iii)determining,basedonthesimilaritybetweenthefeaturesintwosetsoffeatures,whetherthesamplesshouldbedeclaredtobelikelytocomefromthesamesource(“matchingrule”);and

(b)empiricalestimates,fromappropriatelydesignedstudiesfrommultiplegroups,thatestablish(i)themethod’sfalsepositiverate—thatis,theprobabilityitdeclaresaproposedidentificationbetweensamplesthatactuallycomefromdifferentsources,and(ii)themethod’ssensitivity—thatis,theprobabilityitdeclaresaproposedidentificationbetweensamplesthatactuallycomefromthesamesource.

AsdescribedinBox4,scientificvalidationstudiesshouldsatisfyanumberofcriteria:(a)Theyshouldbebasedonsufficientlylargecollectionsofknownandrepresentativesamplesfromrelevantpopulations;(b)theyshouldbeconductedsothattheexamineeshavenoinformationaboutthecorrectanswer;(c)thestudydesignandanalysisplanshouldbespecifiedinadvanceandnotmodifiedafterwardsbasedontheresults;(d)thestudyshouldbeconductedoroverseenbyindividualsororganizationswithnostakeintheoutcome;(e)data,softwareandresultsshouldbeavailabletoallowotherscientiststoreviewthe

159NationalCommissiononForensicScience.RecommendationtotheAttorneyGeneral:RequestforNISTtoEvaluateDevelopmentalValidationStudiesforForensicScienceTestMethodsinAdvanceofDocumentaryStandardsSetting,availableat:https://www.justice.gov/ncfs/file/831536/download


47

conclusions;and(f)toensurethattheresultsarerobustandreproducible,thereshouldbemultipleindependentstudiesbyseparategroupsreachingsimilarconclusions.

Onceamethodhasbeenestablishedasfoundationallyvalidbasedonadequateempiricalstudies,claimsaboutthemethod’saccuracyandtheprobativevalueofproposedidentifications,inordertobevalid,mustbebasedonsuchempiricalstudies.

Forobjectivemethods,foundationalvaliditycanbeestablishedbydemonstratingthereliabilityofeachoftheindividualsteps(featureidentification,featurecomparison,matchingrule,falsematchprobability,andsensitivity).

Forsubjectivemethods,foundationalvaliditycanbeestablishedonlythroughblack-boxstudiesthatmeasurehowoftenmanyexaminersreachaccurateconclusionsacrossmanyfeature-comparisonproblemsinvolvingsamplesrepresentativeoftheintendeduse.Intheabsenceofsuchstudies,asubjectivefeature-comparisonmethodcannotbeconsideredscientificallyvalid.

Foundationalvalidityisasinequanon,whichcanonlybeshownthroughempiricalstudies.Importantly,goodprofessionalpractices—suchastheexistenceofprofessionalsocieties,certificationprograms,accreditationprograms,peer-reviewedarticles,standardizedprotocols,proficiencytesting,andcodesofethics—cannotsubstituteforempiricalevidenceofscientificvalidityandreliability.

(2)Validityasapplied.Onceaforensicfeature-comparisonmethodhasbeenestablishedasfoundationallyvalid,itisnecessarytoestablishitsvalidityasappliedinagivencase.

AsdescribedinBox5,validityasappliedrequiresthat:(a)Theforensicexaminermusthavebeenshowntobecapableofreliablyapplyingthemethod,asshownbyappropriateproficiencytesting(seeSection4.6),andmustactuallyhavedoneso,asdemonstratedbytheproceduresactuallyusedinthecase,theresultsobtained,andthelaboratorynotes,whichshouldbemadeavailableforscientificreviewbyothers;and(b)Assertionsabouttheprobativevalueofproposedidentificationsmustbescientificallyvalid—includingthatexaminersshouldreporttheoverallfalsepositiverateandsensitivityforthemethodestablishedinthestudiesoffoundationalvalidity;demonstratethatthesamplesusedinthefoundationalstudiesarerelevanttothefactsofthecase;whereapplicable,reportprobativevalueoftheobservedmatchbasedonthespecificfeaturesobservedinthecase;andnotmakeclaimsorimplicationsthatgobeyondtheempiricalevidence.


48

5. EvaluationofScientificValidity

forSevenFeature-ComparisonMethodsInthepreviouschapter,wedescribedthescientificcriteriathataforensicfeature-comparisonmethodmustmeettobeconsideredscientificallyvalidandreliable,andweunderscoredtheneedforempiricalevidenceofaccuracyandreliability.

Inthischapter,weillustratethemeaningofthesecriteriabyapplyingthemtosixspecificforensicfeature-comparisonmethods:(1)DNAanalysisofsingle-sourceandsimple-mixturesamples,(2)DNAanalysisofcomplex-mixturesamples,(3)bitemarks,(4)latentfingerprints,(5)firearmsidentification,and(6)footwearanalysis.160Foraseventhforensicfeature-comparisonmethod,hairanalysis,wenotundertakeafullevaluation,butreviewarecentevaluationbytheDOJ.

Weevaluatewhetherthesemethodshavebeenestablishedtobefoundationallyvalidandreliableand,ifso,whatestimatesofaccuracyshouldaccompanytestimonyconcerningaproposedidentification,basedoncurrentscientificstudies.Wealsobrieflydiscusssomeissuesrelatedtovalidityasapplied.

PCASTcompiledalistof2019papersfromvarioussources—includingbibliographiespreparedbytheNationalScienceandTechnologyCouncil’sSubcommitteeonForensicScience,therelevantScientificWorkingGroups(predecessorstothecurrentOSAC),161andtherelevantOSACcommittees;submissionsinresponsetoPCAST’srequestforinformationfromtheforensic-sciencestakeholdercommunity;andourownliteraturesearches.162PCASTmembersandstaffidentifiedandreviewedthosepapersthatwererelevanttoestablishingscientificvalidity.Afterreachingasetofinitialconclusions,inputwasobtainedfromtheFBILaboratoryandindividualscientistsatNIST,aswellasotherexperts—includingaskingthemtoidentifyadditionalpaperssupportingscientificvaliditythatwemighthavemissed.

Foreachofthemethods,weprovideabriefoverviewofthemethodology,discussbackgroundinformationandstudies,andreviewevidenceforscientificvalidity.

AsdiscussedinChapter4,objectivemethodshavewell-definedproceduresto(1)identifythefeaturesinsamples,(2)measurethefeatures,(3)determinewhetherthefeaturesintwosamplesmatchtowithinastatedmeasurementtolerance(matchingrule),and(4)estimatetheprobabilitythatsamplesfromdifferentsourceswouldmatch(falsematchprobability).Itispossibletoexamineeachoftheseseparatestepsfortheirvalidity

160TheAmericanAssociationfortheAdvancementofScience(AAAS)isconductingananalysisoftheunderlyingscientificbasesfortheforensictoolsandmethodscurrentlyusedinthecriminaljusticesystem.AsofAugust,2016noreportshavebeenissued.See:www.aaas.org/page/forensic-science-assessments-quality-and-gap-analysis.161See:www.nist.gov/forensics/workgroups.cfm.162See:www.whitehouse.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensics_references.pdf.


49

andreliability.Ofthesixmethodsconsideredinthischapter,onlythefirsttwomethods(involvingDNAanalysis)employobjectivemethods.Theremainingfourmethodsaresubjective.

Forsubjectivemethods,theproceduresarenotpreciselydefined,butratherinvolvesubstantialexperthumanjudgment.Examinersmayfocusoncertainfeatureswhileignoringothers,maycomparethemindifferentways,andmayhavedifferentstandardsfordeclaringproposedidentificationbetweensamples.AsdescribedinChapter4,thesolewaytoestablishfoundationalvalidityisthroughmultipleindependent“black-box”studiesthatmeasurehowoftenexaminersreachaccurateconclusionsacrossmanyfeature-comparisonproblemsinvolvingsamplesrepresentativeoftheintendeduse.Intheabsenceofsuchstudies,afeature-comparisonmethodcannotbeconsideredscientificallyvalid.

PCASTfoundfewblack-boxstudiesappropriatelydesignedtoassessscientificvalidityofsubjectivemethods.Twonotableexceptions,discussedinthischapter,wereastudyonlatentfingerprintsconductedbytheFBILaboratoryandastudyonfirearmsidentificationsponsoredbytheDepartmentofDefenseandconductedbytheDepartmentofEnergy’sAmesLaboratory.

Weconsideredwhetherproficiencytesting,whichisconductedbycommercialorganizationsforsomedisciplines,couldbeusedtoestablishfoundationalvalidity.Weconcludedthatitcouldnot,atpresent,forseveralreasons.First,proficiencytestsarenotintendedtoestablishfoundationalvalidity.Second,thetestproblemsortestsetsusedincommercialproficiencytestsarenotatpresentroutinelymadepublic—makingitimpossibletoascertainwhetherthetestsappropriatelyassessthemethodacrosstherangeofapplicationsforwhichitisused.Thepublicationandcriticalreviewofmethodsanddataisanessentialcomponentinestablishingscientificvalidity.Third,thedominantcompanyinthemarket,CollaborativeTestingServices,Inc.(CTS),explicitlystatesthatitsproficiencytestsarenotappropriateforestimatingerrorratesofadiscipline,because(a)thetestresults,whichareopentoanyone,maynotreflecttheskillsofforensicpractitioners,and(b)“thereportedresultsdonotreflect‘correct’or‘incorrect’answers,butratherresponsesthatagreeordisagreewiththeconsensusconclusionsoftheparticipantpopulation.”163Fourth,thetestsforforensicfeature-comparisonmethodstypicallyconsistofonlyoneortwoproblemseachyear.Fifth,“easytestsarefavoredbythecommunity”,withtheresultthatteststhataretoochallengingcouldjeopardizerepeatbusinessforacommercialvendor.164,165

PCAST’sobservationsandfindingsbelowarelargelyconsistentwiththeconclusionsofearlierNRCreports.166

163See:www.ctsforensics.com/assets/news/CTSErrorRateStatement.pdf.164PCASTthanksCTSPresidentChristopherCzyrycaforhelpfulconversationsconcerningproficiencytesting.Czyrycaexplainedthatthat(i)CTSdefinesconsensusasatleast80%agreementamongrespondentsand(ii)proficiencytestingforlatentfingerprintsonlyoccasionallyinvolvesaprobleminwhichaquestionedprintmatchesnoneofthepossibleanswers.165“Easytestsarefavoredbythecommunity,”accordingtoapresentationtotheNationalCommissiononForensicSciencebyCTSPresidentCzyryca.See:www.justice.gov/ncfs/file/761061/download.166NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009).NationalResearchCouncil,BallisticImaging.TheNationalAcademiesPress.WashingtonDC.(2008).


50

5.1DNAanalysisofsingle-sourceandsimple-mixturesamples

DNAanalysisofsingle-sourceandsimplemixturesamplesincludesexcellentexamplesofobjectivemethodswhosefoundationalvalidityhasbeenproperlyestablished.

Methodology

DNAanalysisinvolvescomparingDNAprofilesfromdifferentsamplestoseeifaknownsamplemayhavebeenthesourceofanevidentiarysample.

TogenerateaDNAprofile,DNAisfirstchemicallyextractedfromasamplecontainingbiologicalmaterial,suchasblood,semen,hair,orskincells.Next,apredeterminedsetofDNAsegments(“loci”)containingsmallrepeatedsequences167areamplifiedusingthePolymeraseChainReaction(PCR),anenzymaticprocessthatreplicatesatargetedDNAsegmentoverandovertoyieldmillionsofcopies.Afteramplification,thelengthsoftheresultingDNAfragmentsaremeasuredusingatechniquecalledcapillaryelectrophoresis,whichisbasedonthefactthatlongerfragmentsmovemoreslowlythanshorterfragmentsthroughapolymersolution.Therawdatacollectedfromthisprocessareanalyzedbyasoftwareprogramtoproduceagraphicalimage(anelectropherogram)andalistofnumbers(theDNAprofile)correspondingtothesizesoftheeachoffragments(bycomparingthemtoknown“molecularsizestandards”).

Ascurrentlypracticed,themethoduses13specificlociandtheamplificationprocessisdesignedsothattheDNAfragmentscorrespondingtodifferentlocioccupydifferentsizeranges—makingitsimpletorecognizewhichfragmentscomefromeachlocus.168Ateachlocus,everyhumancarriestwovariants(called“alleles”)—oneinheritedfromhisorhermother,onefromhisorherfather—thatmaybeofdifferentlengthsorthesamelength.169

Analysisofsingle-sourcesamplesDNAanalysisofasamplefromasingleindividualisanobjectivemethod.Inadditiontothelaboratoryprotocolsbeingpreciselydefined,theinterpretationalsoinvolveslittleornohumanjudgment.

AnexaminercanassessifasamplecamefromasinglesourcebasedonwhethertheDNAprofiletypicallycontains,foreachlocus,exactlyonefragmentfromeachchromosomecontainingthelocus—whichyieldsoneor

167Therepeats,calledshorttandemrepeats(STRs),consistofconsecutiverepeatedcopiesofasegmentsof2-6basepairs.168ThecurrentkitusedbytheFBI(IdentifilerPlus)has16totalloci:15STRlociandtheamelogeninlocus.Akitthatwillbeimplementedlaterthisyearhas24loci.169TheFBIannouncedin2015thatitplanstoexpandthecorelocibyaddingsevenadditionallocicommonlyusedindatabasesinothercountries.(Populationdatahavebeenpublishedfortheexpandedset,includingfrequenciesin11ethnicpopulationswww.fbi.gov/about-us/lab/biometric-analysis/codis/expanded-fbi-str-2015-final-6-16-15.pdf.)Startingin2017,theselociwillberequiredforuploadingandsearchingDNAprofilesinthenationalsystem.Theexpandeddataineachprofileareexpectedtoprovidegreaterdiscriminationpotentialforidentification,especiallyinmatchingsampleswithonlypartialDNAprofiles,missingpersoninquiries,andinternationallawenforcementandcounterterrorismcases.


51

twodistinctfragmentlengthsfromeachlocus.170TheDNAprofilecanthenbecomparedwiththeDNAprofileofaknownsuspect.ItcanalsobeenteredintotheFBI’sNationalDNAIndexSystem(NDIS)andsearchedagainstadatabaseofDNAprofilesfromconvictedoffenders(andarresteesinmorethanhalfofthestates)orunsolvedcrimes.

TwoDNAprofilesaredeclaredtomatchifthelistsofallelesarethesame.171TheprobabilitythattwoDNAprofilesfromdifferentsourceswouldhavethesameDNAprofile(therandommatchprobability)isthencalculatedbasedontheempiricallymeasuredfrequencyofeachalleleandestablishedprinciplesofpopulationgenetics(seep.53).172

AnalysisofsimplemixturesManysexualassaultcasesinvolveDNAmixturesoftwoindividuals,whereoneindividual(i.e.,thevictim)isknown.DNAanalysisofthesesimplemixturesisalsorelativelystraightforward.Methodshavebeenusedfor30yearstodifferentiallyextractDNAfromspermcellsvs.vaginalepithelialcells,makingitpossibletogenerateDNAprofilesfromthetwosources.Wherethetwocelltypesarethesamebutonecontributorisknown,theallelesoftheknownindividualcanbesubtractedfromthesetofallelesidentifiedinthemixture.173

Oncetheknownsourceisremoved,theanalysisoftheunknownsamplethenproceedsasaboveforsingle-sourcesamples.Liketheanalysisofsingle-sourcesamples,theanalysisofsimplemixturesisalargelyobjectivemethod.

FoundationalValidity

Toevaluatethefoundationalvalidityofanobjectivemethod(suchassingle-sourceandsimplemixtureanalysis),onecanexaminethereliabilityofeachoftheindividualstepsratherthanhavingtorelyonblack-boxstudies.

Single-sourcesamplesEachstepintheanalysisisobjectiveandinvolveslittleornohumanjudgment.

170Theexaminerreviewstheelectropherogramtodeterminewhethereachofthepeaksisatrueallelicpeaksoranartifact(e.g.,backgroundnoiseintheformofstutter,spikes,andotherphenomena)andtodeterminewhethermorethanoneindividualcouldhavecontributedtotheprofile.Inrarecases,anindividualmayhavetwofragmentsatalocusduetorarecopy-numbervariationinthehumangenome.171Whenonlyapartialprofilecouldbegeneratedfromtheevidencesample(forexample,incaseswithlimitedquantitiesofDNA,degradationofthesample,orthepresenceofcontaminants),anexaminermayalsoreportan“inclusion”ifthepartialprofileisconsistentwiththeDNAprofileobtainedfromareferencesample.AnexaminermayalsoreportaninclusionwhentheDNAresultsfromareferencesamplearepresentinamixture.Thesecasesgenerallyrequiresignificantlymorehumananalysisandinterpretationthansingle-sourcesamples.172Randommatchprobabilitiescanalsobeexpressedintermsofalikelihoodratio(LR),whichistheratioof(1)theprobabilityofobservingtheDNAprofileiftheindividualinquestionisthesourceoftheDNAsampleand(2)theprobabilityofobservingtheDNAprofileiftheindividualinquestionisnotthesourceoftheDNAsample.Inthesituationofasingle-sourcesample,theLRshouldbesimplythereciprocaloftherandommatchprobability(becausethefirstprobabilityintheLRis1andthesecondprobabilityistherandommatchprobability).173Inmanycases,DNAwillbepresentinthemixtureinsufficientlydifferentquantitiessothatthepeakheightsintheelectropherogramfromthetwosourceswillbedistinct,allowingtheexaminertomorereadilyseparateoutthesources.


52

(1)Featureidentification.Incontrasttotheothermethodsdiscussedinthisreport,thefeaturesusedinDNAanalysis(thefragmentslengthsoftheloci)aredefinedinadvance.

(2)Featuremeasurementandcomparison.PCRamplification,inventedin1983,iswidelyusedbytensofthousandsofmolecularbiologylaboratories,includingformanymedicalapplicationsinwhichithasbeenrigorouslyvalidated.MultiplexPCRkitsdesignedbycommercialvendorsforusebyforensiclaboratoriesmustbevalidatedbothexternally(throughdevelopmentalvalidationstudiespublishedinpeerreviewedpublication)andinternally(byeachlabthatwishestousethekit)beforetheymaybeused.174Fragmentsizesaremeasuredbyanautomatedprocedurewhosevariabilityiswellcharacterizedandsmall;thestandarddeviationisapproximately0.05basepairs,whichprovideshighlyreliablemeasurements.175,176Developmentalvalidationstudieswereperformed—includingbytheFBI—toverifytheaccuracy,precision,andreproducibilityoftheprocedure.177,178

(3)Featurecomparison.Forsingle-sourcesamples,thereareclearandwell-specified“matchingrules”fordeclaringwhethertheDNAprofilesmatch.WhencompleteDNAprofilesaresearchedagainsttheNDISat“highstringency,”a“match”isreturnedonlywheneachalleleintheunknownprofileisfoundtomatchanalleleoftheknownprofile,andviceversa.WhenpartialDNAprofilesobtainedfromapartially

174LaboratoriesthatconductforensicDNAanalysisarerequiredtofollowFBI’sQualityAssuranceStandardsforDNATestingLaboratoriesasaconditionofparticipatingintheNationalDNAIndexSystem(www.fbi.gov/about-us/lab/biometric-analysis/codis/qas-standards-for-forensic-dna-testing-laboratories-effective-9-1-2011).FBI’sScientificWorkingGrouponDNAAnalysisMethods(SWGDAM)haspublishedguidelinesforlaboratoriesinvalidatingproceduresconsistenttheFBI’sQualityAssuranceStandards(QAS).SWGDAMValidationGuidelinesforDNAAnalysisMethods,December2012.See:media.wix.com/ugd/4344b0_cbc27d16dcb64fd88cb36ab2a2a25e4c.pdf.175ForensiclaboratoriestypicallyusegeneticanalyzersystemsdevelopedbytheAppliedBiosystemsgroupofThermo-FisherScientific(ABI310,3130,or3500).176Toincorrectlyestimateafragmentlengthby1basepair(theminimumsizedifference)requiresameasurementerrorof0.5basepair,whichcorrespondsto10standarddeviations.Moreover,allelestypicallydifferbyatleast4basepairs(althoughsomeSTRlocihavefairlycommonallelesthatdifferby1or2nucleotides).177Forexamplesofthesestudiessee:Budowle,B.,Moretti,T.R.,Keys,K.M.,Koons,B.W.,andJ.B.Smerick.“ValidationstudiesoftheCTTSTRmultiplexsystem.”JournalofForensicSciences,Vol.42,No.4(1997):701-7;Kimpton,C.P.,Oldroyd,N.J.,Watson,S.K.,Frazier,R.R.,Johnson,P.E.,Millican,E.S.,Urguhart,A.,Sparkes,B.L.,andP.Gill.“Validationofhighlydiscriminatingmultiplexshorttandemrepeatamplificationsystemsforindividualidentification.”Electrophoresis,Vol.17,No.8(1996):1283-93;Lygo,J.E.,Johnson,P.E.,Holdaway,D.J.,Woodroffe,S.,Whitaker,J.P.,Clayton,T.M.,Kimpton,C.P.,andP.Gill.“Thevalidationofshorttandemrepeat(STR)lociforuseinforensiccasework.”InternationalJournalofLegalMedicine,Vol.107,No.2(1994):77-89;andFregeau,C.J.,Bowen,K.L.,andR.M.Fourney.“ValidationofhighlypolymorphicfluorescentmultiplexshorttandemrepeatsystemsusingtwogenerationsofDNAsequencers.”JournalofForensicSciences,Vol.44,No.1(1999):133-66.178Forexample,a2001studythatcomparedtheperformancecharacteristicsofseveralcommerciallyavailableSTRtestingkitstestedtheconsistencyandreproducibilityofresultsusingpreviouslytypedcasesamples,environmentallyinsultedsamples,andbodyfluidsamplesdepositedonvarioussubstrates.ThestudyfoundthatallofthekitscouldbeusedtoamplifyandtypeSTRlocisuccessfullyandthattheproceduresusedforeachofthekitswererobustandvalid.Noevidenceoffalsepositiveorfalsenegativeresultsandnosubstantialevidenceofpreferentialamplificationwithinalocuswerefoundforanyofthetestingkits.Moretti,T.R.,Baumstark,A.L.,Defenbaugh,D.A.,Keys,K.M.,Smerick,J.B.,andB.Budowle.“ValidationofShortTandemRepeats(STRs)forforensicusage:performancetestingoffluorescentmultiplexSTRsystemsandanalysisofauthenticandsimulatedforensicsamples.”JournalofForensicSciences,Vol.46,No.3(2001):647-60.


53

degradedorcontaminatedsamplearesearchedat“moderatestringency,”candidateprofilesarereturnedifeachoftheallelesintheunknownprofileisfoundtomatchanalleleoftheknownprofile.179,180

(4)Estimationofrandommatchprobability.Theprocessforcalculatingtherandommatchprobability(thatis,theprobabilityofamatchoccurringbychance)isbasedonwell-establishedprinciplesofpopulationgeneticsandstatistics.ThefrequenciesoftheindividualalleleswereobtainedbytheFBIbasedonDNAprofilesfromapproximately200unrelatedindividualsfromeachofsixpopulationgroupsandwereevaluatedpriortouse.181Thefrequencyofanoverallpatternofalleles—thatis,therandommatchprobability—istypicallyestimatedbymultiplyingthefrequenciesoftheindividualloci,undertheassumptionthattheallelesareindependentofoneanother.182Theresultingprobabilityistypicallylessthan1in10billion,excludingthepossibilityofcloserelatives.183

Thecalculationsometimesoverstatestherarityofapatternbecausetheallelesarenotcompletelyindependent,owingtopopulationsubstructure.A1996NRCreportconcludedthatthetrueprobabilitywaslikelytobewithinafactorof10ofthecalculatedvalue(forexample,forarandommatchprobabilityestimateof1in10million,thetrueprobabilityishighlylikelytobebetween1in1millionand1in100million).184However,

179See:FBI’s:FrequentlyAskedQuestions(FAQs)ontheCODISProgramandtheNationalDNAIndexSystem.www.fbi.gov/about-us/lab/biometric-analysis/codis/codis-and-ndis-fact-sheet.180ContaminatedsamplesarenotsubmittedtoCODIS.181TheinitialpopulationdatageneratedbyFBIincludeddatafor6ethnicpopulationswithdatabasesizesof200individuals.See:Budowle,B.,Moretti,T.R.,Baumstark,A.L.,Defenbaugh,D.A.,andK.M.Keys.“PopulationdataonthethirteenCODIScoreshorttandemrepeatlociinAfricanAmericans,U.S.Caucasians,Hispanics,Bahamians,Jamaicans,andTrinidadians.”JournalofForensicSciences,Vol.44,No.6(1999):1277-86.Budowle,B.,Shea,B.,Niezgoda,S.,andR.Chakraborty.“CODISSTRlocidatafrom41samplepopulations.”JournalofForensicSciences,Vol.46,No.3(2001):453-89.ThedatabaseshavebeenmaintainedbyFBIandcanbefoundatwww.fbi.gov/about-us/lab/forensic-science-communications/fsc/july1999/budowle.htm/dnaloci.txt.ErrorsintheoriginaldatabasewerereportedinJuly2015(Erratum,JournalofForensicSciences,Vol.60,No.4(2015):1114-6),theimpactofthesediscrepanciesonprofileprobabilitycalculationswereassessed(andfoundtobelessthanafactorof2inafullprofile),andtheallelefrequencyestimateswereamendedaccordingly.See:www.fbi.gov/about-us/lab/biometric-analysis/codis/amended-fbi-str-final-6-16-15.pdf.Atthesametimeasamendingtheoriginaldatasets,theFBILaboratoryalsopublishedexpandeddatasetsinwhichtheoriginalsampleswereretypedforadditionalloci.Inaddition,thepopulationsamplesthatwereoriginallystudiedatotherlaboratoriesweretypedforadditionalloci,sothefulldatasetincludes9populations.These“expanded”datasetsareinuseattheFBILaboratoryandcanbefoundatwww.fbi.gov/about-us/lab/biometric-analysis/codis/expanded-fbi-str-final-6-16-15.pdf.182Moreprecisely,thefrequencyateachlocusiscalculatedfirst.Ifthelocushastwocopiesofthesameallelewithfrequencyp,thefrequencyiscalculatedasp2.Ifthelocushastwodifferentalleleswithrespectivefrequenciespandq,thefrequencyiscalculatedas2pq.Thefrequencyoftheoverallpatterniscalculatedbymultiplyingtogetherthevaluesfortheindividualloci.183Therandommatchprobabilitywillbehigherforcloserelatives.Foridenticaltwins,theDNAprofilesareexpectedtomatchperfectly.Forfirstdegreerelatives,therandommatchprobabilitymaybeontheorderof1in100,000whenexaminingthe13CODIScoreSTRloci.See:Butler,J.M.“ThefutureofforensicDNAanalysis.”PhilosophicalTransactionsoftheRoyalSocietyB,370:20140252(2015).184NationalResearchCouncil.TheEvaluationofForensicDNAEvidence.TheNationalAcademiesPress.WashingtonDC.(1996).Goode,M.“SomeobservationsonevidenceofDNAfrequency.”AdelaideLawReview,Vol.23(2002):45-77.


54

thiscorrectionfactorwasbasedonearlier,lessprecisemethodologies.ArecentstudybyNISTscientistssuggeststhatthevariationmaybesubstantiallygreaterthan10-fold.185Effortsareneededtoresolvethisissue.Nonetheless,therandommatchprobabilityisclearlyverylow.

SimplemixturesThestepsforanalyzingsimplemixturesarethesameasforanalyzingsingle-sourcesamples,upuntilthepointofinterpretation.DNAprofilesthatcontainamixtureoftwocontributors,whereonecontributorisknown,canbeinterpretedinmuchthesamewayassingle-sourcesamples.Thisoccursfrequentlyinsexualassaultcases,whereaDNAprofilecontainsamixtureofDNAfromthevictimandtheperpetrator.MethodsthatareusedtodifferentiallyextractDNAfromspermcellsvs.vaginalepithelialcellsinsexualassaultcasesarewell-established.186Wherethetwocelltypesarethesame,oneDNAsourcemaybedominant,resultinginadistinctcontrastinpeakheightsbetweenthetwocontributors.Theallelesfromthemajorcontributor(correspondingtothelargerallelicpeaks)andtheminorcontributorcanusuallybedistinguishedinthesecases.187

ValidityasApplied

WhileDNAanalysisofsingle-sourcesamplesandsimplemixturesisafoundationallyvalidandreliablemethod,itisnotinfallibleinpractice.ErrorscananddooccurinDNAtesting.AlthoughtheprobabilitythattwosamplesfromdifferentsourceshavethesameDNAprofileistiny,thechanceofhumanerrorismuchhigher.Sucherrorsmaystemfromsamplemix-ups,contamination,incorrectinterpretation,anderrorsinreporting.188

Tominimizehumanerror,theFBIrequires,asaconditionofparticipatinginNDIS,thatlaboratoriesfollowtheFBI’sQualityAssuranceStandards(QAS).189BeforetheresultsoftheDNAanalysiscanbecompared,theexaminerisrequiredtorunaseriesofcontrolstocheckforpossiblecontaminationandensurethatthePCRprocessranproperly.TheQASalsorequiressemi-annualproficiencytestingofallDNAanalyststhatperform

185Gittelson,S.andJ.Buckleton.“Isthefactorof10stillapplicabletoday?”Presentationatthe68thAnnualAmericanAcademyofForensicSciencesScientificMeeting,2016.See:www.cstl.nist.gov/strbase/pub_pres/Gittelson-AAFS2016-Factor-of-10.pdf.186Gill,P.,Jeffreys,A.J.,andD.J.Werrett.“ForensicapplicationofDNA‘fingerprints’.”Nature,Vol.318,No.6046(1985):577-9.187Clayton,T.M.,Whitaker,J.P.,Sparkes,R.,andP.Gill.“AnalysisandinterpretationofmixedforensicstainsusingDNASTRprofiling.”ForensicScienceInternational,Vol.91,No.1(1998):55-70.188Krimsky,S.,andT.Simoncelli.GeneticJustice:DNADataBanks,CriminalInvestigations,andCivilLiberties.ColumbiaUniversityPress,(2011).PerhapsthemostspectacularhumanerrortodateinvolvedtheGermangovernment’sinvestigationofthe“PhantomofHeilbronn,”awomanwhoseDNAappearedatthescenesofmorethan40crimesinthreecountries,including6murders,severalmuggingsanddozensofbreak-insoverthecourseofmorethanadecade.AfteraneffortthatincludedanalyzingDNAsamplesfrommorethan3,000womenfromfourcountriesandthatcost$18million,authoritiesdiscoveredthatthewomanofinterestwasaworkerintheAustrianfactorythatfabricatedtheswabsusedinDNAcollection.ThewomanhadinadvertentlycontaminatedalargenumberofswabswithherownDNA,whichwasthusfoundinmanyDNAtests.189FBI.“QualityassurancestandardsforforensicDNAtestinglaboratories.”(2011).See:www.fbi.gov/about-us/lab/biometric-analysis/codis/qas-standards-for-forensic-dna-testing-laboratories-effective-9-1-2011.


55

DNAtestingforcriminalcases.Theresultsofthetestsdonothavetobepublished,butthelaboratorymustretaintheresultsofthetests,anydiscrepanciesorerrorsmade,andcorrectiveactionstaken.190

ForensicpractitionersintheU.S.donottypicallyreportqualityissuesthatariseinforensicDNAanalysis.Bycontrast,errorratesinmedicalDNAtestingarecommonlymeasuredandreported.191Refreshingly,a2014paperfromtheNetherlandsForensicInstitute,agovernmentagency,reportedacomprehensiveanalysisofall“qualityissuenotifications”encounteredincasework,categorizedbytype,sourceandimpact.192,193Theauthorscallforgreater“transparency”and“culturechange,”writingthat:

ForensicDNAcaseworkisconductedworldwideinalargenumberoflaboratories,bothprivatecompaniesandininstitutesownedbythegovernment.Qualityproceduresareinplaceinalllaboratories,butthenatureofthequalitysystemvariesalotbetweenthedifferentlabs.Inparticular,therearemanyforensicDNAlaboratoriesthatoperatewithoutaqualityissuenotificationsystemliketheonedescribedinthispaper.Inourexperience,suchasystemisextremelyimportantforthedetectionandproperhandlingoferrors.Thisiscrucialinforensiccaseworkthatcanhaveamajorimpactonpeople’slives.WethereforeproposethattheimplementationofaqualityissuenotificationsystemisnecessaryforanylaboratorythatisinvolvedinforensicDNAcasework.

Suchsystemcanonlyworkinanoptimalway,however,whenthereisablame-freecultureinthelaboratorythatextendstothepoliceandthelegaljusticesystem.Peoplehaveanaturaltendencytohidetheirmistakes,anditisessentialtocreateanatmospherewheretherearenoadversepersonalconsequenceswhenmistakesarereported.Themanagementshouldtaketheleadinthisculturechange....

Asfarasweknow,theNFIisthefirstforensicDNAlaboratoryintheworldtorevealsuchdetaileddataandreports.Itshowsthatthisispossiblewithoutanydisastersorabusehappening,andtherearenoreasonsfornondisclosure.Asmentionedintheintroduction,inlaboratorymedicinepublicationofdataonerrorrateshasbecomestandardpractice.Qualityfailureratesinthisdomainarecomparabletoours.

Finally,wenotethatthereisaneedtoimproveproficiencytesting.Therearecurrentlynorequirementsconcerninghowchallengingtheproficiencytestsshouldbe.

190Ibid,Sections12,13,and14.191See,forexample:Plebani,M.,andP.Carroro.“Mistakesinastatlaboratory:typesandfrequency.”ClinicalChemistry,Vol.43(1997):1348-51;Stahl,M.,Lund,E.D.,andI.Brandslund.“Reasonsforalaboratory’sinabilitytoreportresultsforrequestedanalyticaltests.”ClinicalChemistry,Vol.44(1998):2195-7;Hofgartner,W.T.,andJ.F.Tait.“Frequencyofproblemsduringclinicalmolecular-genetictesting.”AmericanJournalofClinicalPathology,Vol.112(1999):14-21;andCarroro,P.,andM.Plebani.“Errorsinastatlaboratory:typesandfrequencies10yearslater.”ClinicalChemistry,Vol.53(2007):1338-42.192Kloosterman,A.,Sjerps,M.,andA.Quak.“ErrorratesinforensicDNAanalysis:Definition,numbers,impactandcommunication.”ForensicScienceInternational:Genetics,Vol.12(2014):77-85.J.M.Butler“DNAErrorRates”presentationattheInternationalForensicsSymposium,Washington,D.C.(2015).www.nist.gov/director/upload/Butler-ErrorManagement-DNA-Error.pdf.193TheNetherlandsusesan“inquisitorial”approachtomethodofcriminaljusticeratherthantheadversarialsystemusedintheU.S.ConcernsabouthavingtoexplainqualityissuesincourtmayexplaininpartwhyU.S.laboratoriesdonotroutinelyreportqualityissues.


56

Finding2:DNAAnalysis

Foundationalvalidity.PCASTfindsthatDNAanalysisofsingle-sourcesamplesorsimplemixturesoftwoindividuals,suchasfrommanyrapekits,isanobjectivemethodthathasbeenestablishedtobefoundationallyvalid.

Validityasapplied.Becauseerrorsduetohumanfailureswilldominatethechanceofcoincidentalmatches,thescientificcriteriaforvalidityasappliedrequirethatanexpert(1)shouldhaveundergonerigorousandrelevantproficiencytestingtodemonstratetheirabilitytoreliablyapplythemethod,(2)shouldroutinelydiscloseinreportsandtestimonywhether,whenperformingtheexamination,heorshewasawareofanyfactsofthecasethatmightinfluencetheconclusion,and(3)shoulddisclose,uponrequest,allinformationaboutqualitytestingandqualityissuesinhisorherlaboratory.

5.2DNAanalysisofcomplex-mixturesamples

SomeinvestigationsinvolveDNAanalysisofcomplexmixturesofbiologicalsamplesfrommultipleunknownindividualsinunknownproportions.Suchsamplesmightarise,forexample,frommixedbloodstains.Recently,therehasbeengrowinginterestin“touchDNA”—forexample,tinyquantitiesofDNAleftbymultipleindividualsonasteeringwheelofacar.

Methodology

ThefundamentaldifferencebetweenDNAanalysisofcomplex-mixturesamplesandDNAanalysisofsingle-sourceandsimplemixturesliesnotinthelaboratoryprocessing,butintheinterpretationoftheresultingDNAprofile.

DNAanalysisofcomplexmixtures—definedasmixtureswithmorethantwocontributors—isinherentlydifficultandevenmoreforsmallamountsofDNA.194SuchsamplesresultinaDNAprofilethatsuperimposesmultipleindividualDNAprofiles.Interpretingamixedprofileisdifferentformultiplereasons:eachindividualmaycontributetwo,oneorzeroallelesateachlocus;theallelesmayoverlapwithoneanother;thepeakheightsmaydifferconsiderably,owingtodifferencesintheamountandstateofpreservationoftheDNAfromeachsource;andthe“stutterpeaks”thatsurroundalleles(commonartifactsoftheDNAamplificationprocess)canobscureallelesthatarepresentorsuggestallelesthatarenotpresent.195Itisoftenimpossibletotellwithcertaintywhichallelesarepresentinthemixtureorhowmanyseparateindividualscontributedtothemixture,letaloneaccuratelytoinfertheDNAprofileofeachindividual.196

194 See,e.g.,SWGDAMdocumentoninterpretationofDNAmixtures.http://www.swgdam.org/#!public-comments/c1t82.195Challengeswith“low-template”DNAaredescribedinarecentpaper,Butler,J.M.“ThefutureofforensicDNAanalysis.”PhilosophicalTransactionsoftheRoyalSocietyB,370:20140252(2015).196See:Buckleton,J.S.,Curran,J.M.,andP.Gill.“TowardsunderstandingtheeffectofuncertaintyinthenumberofcontributorstoDNAstains.”ForensicScienceInternationalGenetics,Vol.1,No.1(2007):20-8andCoble,M.D.,Bright,J.A.,


57

Instead,examinersmustask:“Couldasuspect’sDNAprofilebepresentwithinthemixtureprofile?And,whatistheprobabilitythatsuchanobservationmightoccurbychance?”Thequestions(oftencalled“probabilisticgenotyping”)arechallengingforthereasonsgivenabove.BecausemanydifferentDNAprofilesmayfitwithinsomemixtureprofiles,theprobabilitythatasuspect“cannotbeexcluded”asapossiblecontributortocomplexmixturemaybemuchhigher(insomecases,millionsoftimeshigher)thantheprobabilitiesencounteredformatchestosingle-sourceDNAprofiles.

SubjectiveInterpretation

Initialapproachestotheinterpretationofcomplexmixturesreliedonsubjectivejudgmentbyexaminers,togetherwiththeuseofsimplifiedanalyticalapproachessuchas“CombinedProbabilityofInclusion”(CPI)calculations(basedoninclusionandexclusionofalleles).Theseapproachesareproblematicbecausesubjectivechoicesmadebyexaminers,andthelimitationsofthemethods,cansignificantlyaffecttheanswerandtheestimatedprobativevalue—andhaveasignificantriskofbothinaccurateanswersandcognitivebias.

Theproblemwithsubjectiveanalysisofcomplex-mixturesamplesisillustratedbya2003double-homicidecase,Winstonv.Commonwealth.197AprosecutionexpertreportedthatthedefendantcouldnotbeexcludedasapossiblecontributortoDNAonadiscardedglovethatcontainedamixedDNAprofileofatleastthreecontributors;thedefendantwasconvictedandsentencedtodeath.Theprosecutortoldthejurythatthechancethematchoccurredbychancewas1in1.1billion.A2009paper,however,makesareasonablescientificcasethatthatthechanceiscloserto1in2–thatis,50%oftherelevantpopulationcouldnotbeexcluded.198Suchalargediscrepancyisunacceptable,especiallyincaseswhereadefendantwassentencedtodeath.

TwopapersclearlydemonstratethatthesecommonlyusedapproachesforDNAanalysisofcomplexmixturescanbeproblematic.Ina2011study,DrorandHampikiantestedwhetherirrelevantcontextualinformationbiasedtheirconclusionsofexaminers,usingDNAevidencefromanactualadjudicatedcriminalcase(agangrapecaseinGeorgia).199Inthiscase,oneofthesuspectsimplicatedanotherinconnectionwithapleabargain.ThetwoexpertswhoexaminedevidencefromthecrimescenewereawareofthistestimonyagainstthesuspectandknewthatthepleabargaintestimonycouldbeusedincourtonlywithcorroboratingDNAevidence.DuetothecomplexnatureoftheDNAmixturecollectedfromthecrimescene,theanalysisofthisevidencerequiredjudgmentandinterpretationonthepartoftheexaminers.Thetwoexpertsbothconcludedthatthesuspectcouldnotbeexcludedasacontributor.

DrorandHampikianpresentedtheoriginalDNAevidencefromthiscrimeto17expertDNAexaminers,butwithoutanyoftheirrelevantcontextualinformation.Theyfoundthatonly1outofthe17expertsagreedwith

Buckleton,J.S.,andJ.M.Curran.“UncertaintyinthenumberofcontributorsintheproposednewCODISset.”ForensicScienceInternationalGenetics,Vol.19(2015):207-11.197Winstonv.Commonwealth,604S.E.2d21(Va.2004).198Thompson,W.C.“Paintingthetargetaroundthematchingprofile:theTexassharpshooterfallacyinforensicDNAinterpretation.”Law,ProbabilityandRisk,Vol.8,No.3(2009):257-76.199Dror,I.E.,andG.Hampikian.“SubjectivityandbiasinforensicDNAmixtureinterpretation.”Science&Justice,Vol.51,No.4(2011):204-8.


58

theoriginalexpertswhowereexposedtothebiasinginformation(infact,12oftheexaminersexcludedthesuspectasapossiblecontributor).

Ina2016paper,deKeijserandcolleaguespresented19DNAexpertswithamockcaseinvolvinganallegedviolentrobberyoutsideabar:

Thereisamalesuspect,whodeniesanywrongdoing.TheitemsthatweresampledforDNAanalysisaretheshirtofthe(alleged)femalevictim(whoclaimstohavebeengrabbedbyherassailant),acigarettebuttthatwaspickedupbythepoliceandthatwasallegedlysmokedbythevictimand/orthesuspect,andnailclippingsfromthevictim,whoclaimstohavescratchedtheperpetrator.

AlthoughalltheexpertswereprovidedthesameDNAprofiles(preparedfromthethreesamplesaboveandthetwopeople),theirconclusionsvariedwildly.Oneexaminerexcludedthesuspectasapossiblecontributor,whileanotherexaminerdeclaredamatchbetweenthesuspect’sprofileandafewminorpeaksinthemixedprofilefromthenails—reportingarandommatchprobabilityofroughly1in209million.Stillotherexaminersdeclaredtheevidenceinconclusive.200

Asasubjectivemethod,includingwithwidely-usedCombined-Probability-of-Inclusionmethods,DNAanalysisofcomplexmixtureshasnotbeenestablishedtobefoundationallyvalid.201

CurrentEffortstoDevelopObjectiveMethods

Giventheseproblems,severalgroupshavelaunchedeffortstodevelop“probabilisticgenotyping”computerprogramsthatapplyvariousalgorithmstointerpretcomplexmixtures.202AsofMarch2014,atleast8probabilisticgenotypingsoftwareprogramshadbeendeveloped(calledLRmix,LabRetriever,likeLTD,FST,ArmedXpert,TrueAllele,STRmix,andDNAViewMixtureSolution),withsomebeingopensourcesoftwareandsomebeingcommercialproducts.203TheFBILaboratorybeganusingtheSTRmixprogramlessthanayearago,inDecember2015,andisstillintheprocessofpublishingitsowninternaldevelopmentalvalidation.

Theseprobabilisticgenotypingsoftwareprogramsclearlyrepresentamajorimprovementoverpurelysubjectiveinterpretation.However,theystillrequirecarefulscrutinytodetermine(1)whetherthemethodsarescientificallyvalid,includingdefiningthelimitationsontheirreliability(thatis,thecircumstancesinwhichthey

200deKeijser,J.W.,Malsch,M.,Luining,E.T.,Kranenbarg,M.W.,andD.J.H.M.Lenssen.“DifferentialreportingofmixedDNAprofilesanditsimpactonjurists’evaluationofevidence:Aninternationalanalysis.”ForensicScienceInternational:Genetics,Vol.23(2016):71-82.201Prieto,L.,etal.,Euroforgen-NoEcollaborativeexerciseonLRmixtodemonstratestandardizationoftheinterpretationofcomplexDNAprofiles,ForensicSci.Int.Genet.9(2014)47–54.202ThemethodsincludetheCombinedProbabilityofInclusion(CPI),restrictedLikelihoodRatios,andunrestrictedLikelihoodRatios.SeeSWGDAM2010InterpretationGuidelines(www.swgdam.org).Gill,P.,Brenner,C.H.,Buckleton,J.S.,Carracedo,A.,Krawczak,M.,Mayr,W.R.,Morling,N.,Prinz,M.,Schneider,P.M.,andB.S.Weir.“DNAcommissionoftheInternationalSocietyofForensicGenetics:Recommendationsontheinterpretationofmixtures.”ForensicScienceInternational,Vol.160,No.2-3(2006):90-101.203ThetopicisreviewedinButler,J.M."Chapter13:CopingwithPotentialMissingAlleles."AdvancedTopicsinForensicDNATyping:Interpretation.Waltham,MA:Elsevier/Academic,2015.333-48.


59

mayyieldunreliableresults)and(2)whetherthesoftwarecorrectlyimplementsthemethods.204Thisisparticularlyimportantbecausetheprogramsemploydifferentmathematicalalgorithmsandcanyielddifferentresultsforthesamemixtureprofile.205

Appropriateevaluationoftheproposedmethodsshouldconsistofstudiesbymultiplegroups,notassociatedwiththesoftwaredevelopers,thatinvestigatetheperformanceanddefinethelimitationsofprogramsbytestingthemonawiderangeofmixtureswithdifferentproperties.Inparticular,itisimportanttoaddressthefollowingissues:

(1) Howwelldoesthemethodperformasafunctionofthenumberofcontributorstothemixture?Howwelldoesitperformwhenthenumberofcontributorstothemixtureisunknown?

(2) Howdoesthemethodperformasafunctionofthenumberofallelessharedamongindividualsinthemixture?Relatedly,howdoesitperformwhenthemixturesincludesrelatedindividuals?

(3) Howwelldoesthemethodperform—andhowdoesaccuracydegrade—asafunctionoftheabsoluteandrelativeamountsofDNAfromthevariouscontributors?Forexample,itcanbedifficulttodeterminewhetherasmallpeakinthemixtureprofilerepresentsatrueallelefromaminorcontributororastutterpeakfromanearbyallelefromadifferentcontributor.(Notably,thisissueunderliesacurrentcasethathasreceivedconsiderableattention.206)

(4) Underwhatcircumstances—andwhy—doesthemethodproduceresults(randominclusionprobabilities)thatdiffersubstantiallyfromthoseproducedbyothermethods?

Atpresent,mostofthestudiesevaluatingthesoftwarepackageshavebeenundertakenbythesoftwaredevelopersthemselves;theyarethusnotindependentscientificevaluations.207Moreover,thestudiesaddress

204Forexample,authoritiesinQueensland,AustraliareportedinMarch2015thatsoftwareerrorsintheSTRmixprogramhadledtothereportingofincorrectresults.http://www.couriermail.com.au/news/queensland/queensland-authorities-confirm-miscode-affects-dna-evidence-in-criminal-cases/news-story/833c580d3f1c59039efd1a2ef55af92b205Someprogramsusediscrete(semi-continuous)methods,whichuseonlyalleleinformationinconjunctionwithprobabilitiesofdrop-outanddrop-in,whileotherprogramsusecontinuousmethods,whichalsoincorporateinformationaboutpeakheightandotherinformation.Withinthesetwoclasses,theprogramsdifferwithrespecttohowtheyusetheinformation.SomeofthemethodsinvolvemakingassumptionsaboutthenumberofindividualscontributingtotheDNAprofile,andusethisinformationtocleanupnoise(suchas“stutter”inDNAprofiles).206McKinley,J.(2016,July24).PotsdamBoy’sMurderCaseMayHingeonMinusculeDNASampleFromFingernail.NewYorkTimes.RetrievedAugust22,2016,fromhttp://www.nytimes.com/2016/07/25/nyregion/potsdam-boys-murder-case-may-hinge-on-statistical-analysis.html207E.g.:Perlin,M.W.,Belrose,J.L.,andDuceman,B.W.NewYorkStateTrueAllele®Caseworkvalidationstudy.JournalofForensicSciences,58(6):1458-1466,2013.;Perlin,M.W.,Dormer,K.,Hornyak,J.,Schiermeier-Wood,L.,andGreenspoon,S.TrueAlleleCaseworkonVirginiaDNAmixtureevidence:computerandmanualinterpretationin72reportedcriminalcases.PLoSONE,9(3):e92837,2014.;Perlin,M.W.,Hornyak,J.M.,Sugimoto,G.,andMiller,K.W.P.TrueAllelegenotypeidentificationonDNAmixturescontaininguptofiveunknowncontributors.JournalofForensicSciences,60(4):857-868,2015.;D.Taylor,J.A.Bright,J.S.Buckleton,TheinterpretationofsinglesourceandmixedDNAprofiles,ForensicScienceInternational:Genetics.7(5)(2013)516-528.;J.A.Bright,D.Taylor,J.M.Curran,J.S.Buckleton,Developingallelicand


60

onlyalimitedsubsetofthekeyissues.208,209And,therehavebeenfewcomparativestudiesacrossthemethodstoevaluatethedifferences—and,toourknowledge,nocomparativestudiesconductedbyindependentgroups.210

Conclusion

Basedonitsevaluationofthepublishedliteraturetodate,PCASTconcludesthat(1)subjectiveanalysisofcomplexDNAmixtures,includingwiththewidely-usedCombined-Probability-of-Inclusionmethods,isnotfoundationallyvalidand(2)objectiveanalysisofcomplexDNAmixtureswithprobabilisticgenotypingsoftwareispromising,buthasnotyetbeensufficientlyandappropriatelyvalidatedandtheirlimitationstobeconsideredreliableforallcomplexmixtures.However,reliabilityhasbeenreasonablydemonstratedwithincertainranges(andappropriatestudiesarelikelytoextendtheserangesovertime).Inadditiontoformingitsownjudgment,PCASTalsoconsultedwithJohnButler,SpecialAssistanttotheDirectorforForensicScienceatNISTandViceChairoftheNCFS.211ButlerconcurredwithPCAST’sfinding.

Finding3:DNAanalysisofcomplex-mixturesamples

Foundationalvalidity.PCASTfindsthat:

(1)SubjectiveanalysisofcomplexDNAmixtures,includingwiththewidely-usedCombined-Probability-of-Inclusionmethods,hasnotbeenestablishedtobefoundationallyvalid.

(2)ObjectiveanalysisofcomplexDNAmixtureswithprobabilisticgenotypingsoftwareisrelativelynewandpromisingapproach.Beforethemethodcanbeestablishedasfoundationallyvalidforabroadrangeofsettings,moreresearchisrequiredappropriatelytoestablishthecapabilitiesandlimitationsofvarious

stutterpeakheightmodelsforacontinuousmethodofDNAinterpretation,ForensicScienceInternational:Genetics.7(2)(2013)296-304.;Bright,J.A.etal.“DevelopmentalvalidationofSTRmix™,expertsoftwarefortheinterpretationofforensicDNAprofiles.”ForensicSciIntGenet.2016Jul;23:226-39.208ArecentlypublishedstudybythedevelopersoftheSTRmixmethod,recentlyadoptedbytheFBILaboratory,illustratestheissue,see:Bright,J.A.etal.“DevelopmentalvalidationofSTRmix™,expertsoftwarefortheinterpretationofforensicDNAprofiles.”ForensicSciIntGenet.2016Jul;23:226-39.Thestudyincludesonly31samples,withonlysixsampleswithmorethanthreecontributors,onlythreeinwhichtheratiobetweenthemajorandminorcontributorsexceeds5:1,andnonewheretheratioexceeds10:1.Theanalysisassumesthatthenumberofsamplesisknownanddoesnotinvestigateperformanceunderalternativehypotheses.209GreenspoonSA,Schiermeier-WoodL,JenkinsBC.EstablishingthelimitsofTrueAllele®Casework:Avalidationstudy.JForensicSci.2015;60:1263–76.210BilleTW,WeitzSM,CobleMD,BuckletonJ,BrightJ-A.ComparisonoftheperformanceofdifferentmodelsfortheinterpretationoflowlevelmixedDNAprofiles.Electrophoresis.2014;35:3125–33.211ButlerisaworldauthorityonforensicDNAanalysis,whosePh.D.research,conductedattheFBILaboratory,pioneeredtechniquesofmodernforensicDNAanalysisandwhohaswrittenfivewidelyacclaimedtextbooksonforensicDNAtyping.See:Butler,J.M.(2001)ForensicDNATyping:BiologyandTechnologybehindSTRMarkers.AcademicPress,London.;Butler,J.M.(2005)ForensicDNATyping:Biology,Technology,andGeneticsofSTRMarkers(2ndEdition).ElsevierAcademicPress,NewYork.;Butler,J.M.(2010)FundamentalsofForensicDNATyping.ElsevierAcademicPress,SanDiego.;Butler,J.M.(2012)AdvancedTopicsinForensicDNATyping:Methodology.ElsevierAcademicPress,SanDiego.;Butler,J.M.(2015)AdvancedTopicsinForensicDNATyping:Interpretation.ElsevierAcademicPress,SanDiego.


61

approaches.Atpresent,publishedpaperssupportthefoundationalvalidityofanalysis,withsomeprograms,ofDNAmixturesof3individualsinwhichthecontributorinquestionconstitutesatleast20%oftheintactDNAinthemixture.

ThePathForward

ThereisaclearpathtocompletingstudiestoevaluatethefoundationalvalidityoftheproposedmethodsforinterpretationofcomplexDNAmixtures.Theseeffortswillbeaidedbythecreationanddissemination(underappropriatedata-useanddata-privacyrestrictions)oflargecollectionsofhundredsofDNAprofilescreatedfromknownmixtures—representingwidelyvaryingcomplexitywithrespectto(1)thenumberofcontributors,(2)therelationshipsamongcontributors,(3)theabsoluteandrelativeamountsofmaterials,and(4)thestateofpreservationofmaterials—thatcanbeusedbyindependentgroupstoevaluateandcomparethemethods.Notably,thePROVEDItInitiative(ProjectResearchOpennessforValidationwithExperimentalData)atBostonUniversityhasmadeavailablearesourceof25,000profilesfromDNAmixtures.212,213Inadditiontoscientificstudiesoncommonsetsofsamplesforthepurposeofevaluatingfoundationalvalidity,individualforensiclaboratorieswillwanttoconducttheirowninternaldevelopmentalvalidationstudiestoassessthevalidityofthemethodintheirownhands.214

PCASTbelievesthatsuchstudiescanbecompletedwithintwoyears.NISTshouldplayaleadershiproleinthisprocess,byensuringthecreationanddisseminationofmaterialsandstimulatingstudiesbyindependentgroupsthroughgrants,contractsandprizes;andbyevaluatingtheresultsofthesestudies.

5.3BitemarkAnalysisMethodology

Bitemarkanalysisisasubjectivemethod.Ittypicallyinvolvesexaminingmarksleftonavictimoranobjectatthecrimescene,andcomparingthosemarkswithdentalimpressionstakenfromasuspect.215Bitemarkcomparisonisbasedonthepremisesthat(1i)dentalcharacteristics,particularlythearrangementofthefrontteeth,differsubstantiallyamongpeopleand(2ii)skin(orsomeothermarkedsurfaceatacrimescene)canreliablycapturethesedistinctivefeatures.

212www.bu.edu/dnamixtures.213ThecollectioncontainsDNAsampleswith1-to5-personDNAmixtures,amplifiedwithtargetsrangingfrom1to0.007ng.Inthemulti-personmixtures,theratioofcontributorsrangefrom1:1to1:19.Additionally,theprofilesweregeneratedusingavarietyoflaboratoryconditionsfromsamplescontainingpristineDNA;UVdamagedDNA;enzymaticallyorsonicallydegradedDNA;andinhibitedDNA.214TheFBILaboratoryhasrecentlycompletedadevelopmentalvalidationstudyandispreparingitforpublication.215Lessfrequently,marksarefoundonasuspectedperpetratorthatmayhavecomefromavictim.


62

Bitemarkanalysisbeginswithanexaminerdecidingwhetheraninjuryisamarkcausedbyhumanteeth.216Ifso,theexaminercreatesphotographsorimpressionsofthequestionedbitemarkandofthesuspect’sdentition;comparesthebitemarkandthedentition;anddeterminesifthedentition(1)cannotbeexcludedashavingmadethebitemark;(2)canbeexcludedashavingmadethebitemark;or(3)isinconclusive.Thebitemarkstandardsdonotprovidewell-definedstandardsconcerningthedegreeofsimilaritythatmustbeidentifiedtosupportareliableconclusionthatthemarkcouldhaveorcouldnothavebeencreatedbythedentitioninquestion.Conclusionsaboutallthesemattersarelefttotheexaminer’sjudgment.

BackgroundStudies

Beforeturningtothequestionoffoundationalvalidity,wediscusssomebackgroundstudies(concerningsuchtopicsasuniquenessandconsistency)thatshedsomelightonthefield.Thesestudiescastseriousdoubtonthefundamentalpremisesofthefield.

Awidelycited1984paperclaimedthat“humandentitionwasuniquebeyondanyreasonabledoubt.”217Thestudyexamined397bitemarkscarefullymadeinawaxwafer,measured12parametersfromeach,and—assuming,withoutanyevidence,thattheparameterswereuncorrelatedwitheachother—suggestedthatthechanceoftwobitemarkshavingthesameparametersislessthanoneinsixtrillion.Thepaperwastheoreticalratherthanempirical:itdidnotattempttoactuallycomparethebitemarkstooneanother.

A2010paperdebunkedtheseclaims.218Byempiricallystudying344humandentalcastsandmeasuringthembythree-dimensionallaserscanning,theseauthorsshowedthatmatchesoccurredvastlymoreoftenthanexpectedunderthetheoreticalmodel.Forexample,thetheoreticalmodelpredictedthattheprobabilityoffindingevenasinglefive-toothmatchamongthecollectionofbitemarksislessthanoneinonemillion;yet,theempiricalcomparisonrevealed32suchmatches.

Notably,thesestudiesexaminedhumandentitionpatternsmeasuredunderidealizedconditions.Bycontrast,skinhasbeenshowntobeanunreliablemediumforrecordingtheprecisepatternofteeth.Studiesthathaveinvolvedinflictingbitemarkseitheronlivingpigs219(usedasamodelofhumanskin)orhumancadavers220havedemonstratedsignificantdistortioninalldirections.A2010studyofexperimentallycreatedbitemarksproducedbyknownbitersconcludedthatskindeformationdistortsbitemarkssosubstantiallyandsovariably

216ABFOBitemarkMethodologyStandardsandGuidelines,abfo.org/wp-content/uploads/2016/03/ABFO-Bitemark-Standards-03162016.pdf(accessedJuly2,2016).217Rawson,R.D.,Ommen,R.K.,Kinard,G.,Johnson,J.,andA.Yfantis.“Statisticalevidencefortheindividualityofthehumandentition.”JournalofForensicSciences,Vol.29,No.1(1984):245-53.218Bush,M.A.,Bush,P.J.,andD.H.Sheets.“Statisticalevidenceforthesimilarityofthehumandentition.”JournalofForensicSciences,Vol.56,No.1(2011):118-23.219Dorion,R.B.J.,ed.BitemarkEvidence:AColorAtlasandText.2nded.CRCPress-Taylor&Francis,BocaRaton,Florida(2011).220Sheets,H.D.,Bush,P.J.,andM.A.Bush.“Bitemarks:distortionandcovariationofthemaxillaryandmandibulardentitionasimpressedinhumanskin.”ForensicScienceInternational,Vol.223,No.1-3(2012):202-7.Bush,M.A.,Miller,R.G.,Bush,P.J.,andR.B.Dorion.“Biomechanicalfactorsinhumandermalbitemarksinacadavermodel.”JournalofForensicSciences,Vol.54,No.1(2009):167-76.


63

thatcurrentproceduresforcomparingbitemarksareunabletoreliablyexcludeorincludeasuspectasapotentialbiter(“Thedataderivedshowednocorrelationandwasnotreproducible,thatis,thesamedentitioncouldnotcreateameasurableimpressionthatwasconsistentinalloftheparametersinanyofthetestcircumstances.”)221Suchdistortionisfurthercomplicatedinthecontextofcriminalcases,wherebitingoftenoccursduringstruggles,inwhichskinmaybestretchedandcontortedatthetimeabitemarkiscreated.

Empiricalresearchsuggeststhatforensicodontologistsdonotconsistentlyagreeevenonwhetheraninjuryisahumanbitemarkatall.AstudybytheAmericanBoardofForensicOdontology(AFBO)222involvedshowingphotosof100patternedinjuriestoABFOboard-certifiedbitemarkanalysts,andaskingthemtoanswerthreebasicquestionsconcerning(1)whethertherewassufficientevidencetorenderanopinionastowhetherthepatternedinjuryisahumanbitemark;(2)whetherthemarkisahumanbitemark,suggestiveofahumanbitemark,ornotahumanbitemark;and(3)whetherdistinctfeatures(archesandtoothmarks)wereidentifiable.223Amongthe38examinerswhocompletedthestudy,itwasreportedthattherewasunanimousagreementonthefirstquestioninonly4ofthe100casesandagreementofatleast90percentinonly20ofthe100cases.Acrossallthreequestions,therewasagreementofatleast90percentinonly8ofthe100cases.

InasimilarstudyinAustralia,15odontologistswereshownaseriesof6bitemarksfromcontemporarycases,fiveofwhichweremarksconfirmedbylivingvictimstohavebeencausedbyteeth,andwereaskedtoexplain,innarrativeform,whethertheinjurieswere,infact,bitemarks.224Thestudyfoundwidevariabilityamongthepractitionersintheirconclusionsabouttheorigin,circumstance,andcharacteristicsofthepatternedinjuryforallsiximages.Surprisingly,thosewiththemostexperience(21ormoreyears)tendedtohavethewidestrangeofopinionsastowhetheramarkwasofhumandentaloriginornot.225Examiners’opinionsvariedconsiderablyastowhethertheythoughtagivenmarkwassuitableforanalysis,andindividualpractitionersdemonstratedlittleconsistencyintheirapproachinanalyzingonebitemarktothenext.Thestudyconcludedthatthis“inconsistencyindicatesafundamentalflawinthemethodologyofbitemarkanalysisandshouldleadtoconcernsregardingthereliabilityofanyconclusionsreachedaboutmatchingsuchabitemarktoadentition.”226

221Bush,M.A.,Cooper,H.I.,andR.B.Dorion.“Inquiryintothescientificbasisforbitemarkprofilingandarbitrarydistortioncompensation.”JournalofForensicSciences,Vol.55,No.4(2010):976-83.222AdamFreemanandIainPretty“ConstructvalidityofBitemarkassessmentsusingtheABFOdecisiontree,”presentationatthe2016AnnualMeetingoftheAmericanAcademyofForensicSciences.See:online.wsj.com/public/resources/documents/ConstructValidBMdecisiontreePRETTYFREEMAN.pdf.223Therawdataaremadeavailablebytheauthorsuponrequest.TheywerereviewedbyProfessorKarenKafadar,amemberofthepanelofSeniorAdvisorsforthisstudy.224Page,M.,Taylor,J.,andM.Blenkin.“Expertinterpretationofbitemarkinjuries–acontemporaryqualitativestudy.”JournalofForensicSciences,Vol.58,No.3(2013):664-72.225Forexample,oneexaminerexpressedcertaintythatoneoftheimageswasabitemark,stating,“Iknowfromexperiencethatthat’steethbecauseIdidacaseatthebeginningoftheyear,thatwhenIfirstlookedattheimagesIdidn’tthinktheywereteeth,becausetheinjuriesweresosevere.ButwhenIsawthemodels,andscratchedthemdownmyarm,theylookedjustlikethat.”Anotherexpresseddoubtthatthesameimagewasabitemark,alsobasedonhisorherexperience:“HonestlyIdon’tthinkit’sabitemark…therecouldbeanynumberofthingsthatcouldhavecausedthat.WhetherthisisindividualtoothmarkshereIdoubt.I’veneverseenanythinglikethat.”Ibid.,at666.226Ibid.,at670.


64

StudiesofScientificValidityandReliability

Asdiscussedabove,thefoundationalvalidityofasubjectivemethodcanonlybeestablishedthroughmultipleindependentblack-boxstudies.

The2009NRCreportfoundthatthescientificvalidityofbitemarkanalysishadnotbeenestablished.227InitsownreviewoftheliteraturePCASTfoundfewempiricalstudiesthatattemptedtostudythevalidityandreliabilityofthemethodstoidentifythesourceofabitemark.

Ina1975paper,twoexaminerswereaskedtomatchphotographsofbitemarksmadeby24volunteersinskinfromfreshlyslaughteredpigswithdentalmodelsfromthesesamevolunteers.228Thephotographsweretakenat0,1,and24hoursafterthebitemarkwasproduced.Examiners’performancewaspooranddeterioratedwithtimefollowingthebite.Theproportionofphotographsincorrectlyattributedwas28%,65%,and84%atthe0,1,and24hourtimepoints.

Ina1999paper,29forensicdentalexperts—aswellas80others,includinggeneraldentists,dentalstudents,andlayparticipants—wereshowncolorprintsofhumanbitemarksfrom50courtcasesandaskedtodecidewhethereachbitemarkwasmadebyanadultorachild.229Thedecisionswerecomparedtotheverdictfromthecases.Allgroupsperformedpoorly.230

Ina2001paper,32AFBO-certifieddiplomateswereaskedtoreporttheircertaintythat4specificbitemarksmighthavecomefromeachof7dentalmodels,consistingofthefourcorrectsourcesandthreeunrelatedsamples.231,232Sucha“closed-set”design(wherethecorrectsourceispresentforeachquestionedsamples)isinappropriateforassessingreliability,becauseitwilltendtounderestimatethefalsepositiverate.233Evenwith

227“Thereiscontinuingdisputeoverthevalueandscientificvalidityofcomparingandidentifyingbitemarks.”NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.151.228Whittaker,D.K.“Somelaboratorystudiesontheaccuracyofbitemarkcomparison.”InternationalDentalJournal,Vol.25,No.3(1975):166–71.229Whittaker,D.K.,Brickley,M.R.,andL.Evans.“Acomparisonoftheabilityofexpertsandnon-expertstodifferentiatebetweenadultandchildhumanbitemarksusingreceiveroperatingcharacteristic(ROC)analysis.”ForensicScienceInternational,Vol.92,No.1(1998):11-20.230Theauthorsaskedobserverstoindicatehowcertaintheywereabitemarkwasmadebyanadult,usinga6pointscale.Receiver-OperatorCharacteristic(ROC)curveswerederivedfromthedata.TheAreaundertheCurve(AUC)wascalculatedforeachgroup(whereAUC=1representsperfectclassificationandAUC=0.5isequivalenttorandomdecision-making).TheAreaundertheCurve(AUC)wasbetween0.62-0.69,whichispoor.231Arheart,K.L.,andI.A.Pretty.“Resultsofthe4thAFBOBitemarkWorkshop-1999.”ForensicScienceInternational,Vol.124,No.2-3(2001):104-11.232Thefourbitemarksconsistedofthreefromcriminalcasesandoneproducedbyanindividualdeliberatelybitingintoablockofcheese.Thesevendentalmodelscorrespondedtothethreedefendantsconvictedinthecriminalcases(presumedtobethebiters),theindividualwhobitthecheese,andthreeunrelatedindividuals.233Inclosed-settests,examinerswillperformwellaslongastheychoosetheclosestmatchingdentalmodel.Inanopen-setdesigninwhichnoneofmodelsmaybecorrect,theopportunityforfalsepositivesishigher.Theopen-setdesignresemblestheapplicationincasework.Seethediscussionbelowconcerningfirearmsanalysis.


65

thisclosed-setdesign,11%ofcomparisonstotheincorrectsourceweredeclaredtobe“probable,”“possible,”or“reasonablemedicalcertainty”matches.

Inanother2001paper,10AFBO-certifieddiplomatesweregiven10independenttests,eachconsistingofbitemarkevidenceandtwopossiblesources.Theevidencewasproducedbyclampingadentalmodelontofreshlyslaughteredpigs,subjectivelyconfirmingthat“sufficientdetailwasrecorded,”andphotographingthebitemark.Thecorrectsourcewaspresentinallbuttwoofthetests(mostlyclosed-setdesign).Themeanfalsepositiveratewas15.9%—thatis,roughly1in6.

Ina2010paper,29examinerswithvariouslevelsoftraining(including9AFBO-certifieddiplomates)wereprovidedwithphotographsof18humanbitemarksanddentitionfromthreehumanindividuals(A,B,C),andwereaskedtodecidewhetherthebitemarkscamefromA,B,C,ornoneoftheabove.Thebitemarkshadbeenproducedinlivepigs,usingabitingmachinewithdentitionfromindividualsA,B,andD(forwhichthedentitionwasnotprovidedtotheexaminers).ForbitemarksproducedbyD,thediplomateserroneouslydeclaredamatchtoA,B,orCin17%ofcases—again,roughly1in6.

Conclusion

Fewempiricalstudieshavebeenundertakentostudytheabilityofexaminerstoaccuratelyidentifythesourceofabitemark.Ofthese,severalemployinappropriatedesignsthatarelikelytooverestimateaccuracy.Theobservedfalsepositiveratesweresohighthatthemethodisclearlyscientificallyunreliableatpresent.

Finding4:Bitemarkanalysis

Foundationalvalidity. PCASTfindsthatbitemarkanalysisdoesnotmeetthescientificstandardsforfoundationalvalidity,andisfarfrommeetingsuchstandards.Tothecontrary,availablescientificevidencestronglysuggeststhatexaminerscannotconsistentlyagreeonwhetheraninjuryisahumanbitemarkandcannotidentifythesourceofbitemarkwithreasonableaccuracy.

ThePathForward

Somepractitionershaveexpressedconcernthattheexclusionofbitemarksincourtcouldhampereffortstoconvictdefendantsinsomecases.234Ifso,thecorrectsolution,fromascienticperspective,wouldnotbetoadmitexperttestimonybasedoninvalidandunreliablemethods,butrathertoattempttodevelopscientificallyvalidmethods.

However,PCASTconsiderstheprospectsofdevelopingbitemarkanalysisintoascientificallyvalidmethodtobelow.Weadviseagainstdevotingsignificantresourcestosuchefforts.

234Thepreciseproportionofcasesinwhichbitemarksplayakeyroleisunclear,butisclearlysmall.


66

5.4LatentFingerprintAnalysisLatentfingerprintanalysiswasfirstproposedforuseincriminalidentificationinthe1800sandhasbeenusedformorethanacentury.Themethodwaslonghailedasinfallible,despitethelackofappropriatestudiestoassessitserrorrate.Asdiscussedabove,thisdearthofempiricaltestingindicatedaseriousweaknessinthescientificcultureofforensicscience—wherevaliditywasassumedratherthanproven.Citingearlierguidelinesnowacknowledgedtohavebeeninappropriate,235theDOJrecentlynoted,

Historically,itwascommonpracticeforanexaminertotestifythatwhenthe…methodologywascorrectlyapplied,itwouldalwaysproducethecorrectconclusion.Thusanyerrorthatoccurredwouldbehumanerrorandtheresultingerrorrateofthemethodologywouldbezero.ThisviewwasdescribedbytheDepartmentofJusticein1984inthepublicationTheScienceofFingerprints,whereitstates,“Ofallthemethodsofidentification,fingerprintingalonehasprovedtobebothinfallibleandfeasible.”236

Inresponsetothe2009NRCreport,thelatentprintanalysisfieldhasmadeprogressinrecognizingtheneedtoperformempiricalstudiestoassessfoundationalvalidityandmeasurereliability.MuchcreditgoestotheFBILaboratory,whichhasledthewayinperformingbothblack-boxstudies,designedtomeasurereliability,and“white-boxstudies,”designedtounderstandthefactorsthataffectexaminers’decisions.237PCASTapplaudstheFBI’sefforts.Therearealsonascenteffortstobegintomovethefieldfromapurelysubjectivemethodtowardanobjectivemethod—althoughthereisstillaconsiderablewaytogotoachievethisimportantgoal.

Methodology

Latentfingerprintanalysistypicallyinvolvescomparing(1)a“latentprint”(acompleteorpartialfriction-ridgeimpressionfromanunknownsubject)thathasbeendevelopedorobservedonanitem)with(2)oneormore“knownprints”(fingerprintsdeliberatelycollectedunderacontrolledsettingfromknownsubjects;alsoreferredtoas“tenprints”),toassesswhetherthetwomayhaveoriginatedfromthesamesource.(Itmayalsoinvolvecomparinglatentprintswithoneanother.)

235FederalBureauofInvestigation.TheScienceofFingerprints.U.S.GovernmentPrintingOffice.(1984):p.iv.236See:www.justice.gov/olp/file/861906/download.237See:Hicklin,R.A.,Buscaglia,J.,Roberts,M.A.,Meagher,S.B.,Fellner,W.,Burge,M.J.,Monaco,M.,Vera,D.,Pantzer,L.R.,Yeung,C.C.,andN.Unnikumaran.“Latentfingerprintquality:asurveyofexaminers.”JournalofForensicIdentification.Vol.61,No.4(2011):385-419;Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Assessingtheclarityoffrictionridgeimpressions.”ForensicScienceInternational,Vol.226,No.1(2013):106-17;Ulery,B.T.,Hicklin,R.A.,Kiebuzinski,G.I.,Roberts,M.A.,andJ.Buscaglia.“Understandingthesufficiencyofinformationforlatentfingerprintvaluedeterminations.”ForensicScienceInternational,Vol.230,No.1-3(2013):99-106;Ulery,B.T.,Hicklin,R.A.,andJ.Buscaglia.“Repeatabilityandreproducibilityofdecisionsbylatentfingerprintexaminers.”PLoSONE,(2012);andUlery,B.T.,Hicklin,R.A.,Roberts,M.A.,andJ.Buscaglia.“Changesinlatentfingerprintexaminers’markupbetweenanalysisandcomparison.”ForensicScienceInternational,Vol.247(2015):54-61.


67

Itisimportanttodistinguishlatentprintsfromknownprints.Aknownprintcontainsfingerprintimagesofuptotenfingerscapturedinacontrolledsetting,suchasanarrestorabackgroundcheck.238Becauseknownprintstendtobeofhighquality,theycanbesearchedautomaticallyandreliablyagainstlargedatabases.Bycontrast,latentprintsincriminalcasesareoftenincompleteandofvariablequality(smudgedorotherwisedistorted),withqualityandclaritydependingonsuchfactorsasthesurfacetouchedandthemechanicsoftouch.

Anexaminermightbecalleduponto(1)comparealatentprinttothefingerprintsofaknownsuspectthathasbeenidentifiedbyothermeans(“identifiedsuspect”)or(2)searchalargedatabaseoffingerprintstoidentifyasuspect(“databasesearch”).

Examinerstypicallyfollowanapproachcalled“ACE”or“ACE-V,”forAnalysis,Comparison,Evaluation,andVerification.239,240Theapproachcallsonexaminerstomakeaseriesofsubjectiveassessments.Anexaminerusessubjectivejudgmenttoselectparticularregionsofalatentprintforanalysis.Iftherearenoidentifiedpersonsofinterest,theexaminerwillrunthelatentprintagainstanAutomatedFingerprintIdentificationSystem(AFIS),241containinglargenumbersofknownprints,whichusesnon-public,proprietaryimage-recognitionalgorithms242togeneratealistofpotentialcandidatesthatsharesimilarfingerprintfeatures.243Theexaminerthenmanuallycomparesthelatentprinttothefingerprintsfromthespecificpersonofinterestor

238See:CommitteeonScience,SubcommitteeonForensicScienceoftheNationalScienceandTechnologyCouncil.“AchievingInteroperabilityforLatentFingerprintIdentificationintheUnitedStates.”(2014).www.whitehouse.gov/sites/default/files/microsites/ostp/NSTC/afis_10-20-2014_draftforcomment.pdf.239“AlatentprintexaminationusingtheACE-Vprocessproceedsasfollows:Analysisreferstoaninitialinformation-gatheringphaseinwhichtheexaminerstudiestheunknownprinttoassessthequalityandquantityofdiscriminatingdetailpresent.Theexaminerconsidersinformationsuchassubstrate,developmentmethod,variouslevelsofridgedetail,andpressuredistortions.Aseparateanalysisthenoccurswiththeexemplarprint.Comparisonistheside-by-sideobservationofthefrictionridgedetailinthetwoprintstodeterminetheagreementordisagreementinthedetails.IntheEvaluationphase,theexaminerassessestheagreementordisagreementoftheinformationobservedduringAnalysisandComparisonandformsaconclusion.Verificationinsomeagenciesisareviewofanexaminer’sconclusionswithknowledgeofthoseconclusions;inotheragencies,itisanindependentre-examinationbyasecondexaminerwhodoesnotknowtheoutcomeofthefirstexamination.”NationalInstituteofStandardsandTechnology.“LatentPrintExaminationandHumanFactors:ImprovingthePracticethroughaSystemsApproach.”(2012),availableat:www.nist.gov/oles/upload/latent.pdf.240Reznicek,M.,Ruth,R.M.,andD.M.Schilens.“ACE-Vandthescientificmethod.”JournalofForensicIdentification,Vol.60,No.1(2010):87-103.241StateandlocaljurisdictionsbeganpurchasingAFISsystemsinthe1970sand1980sfromprivatevendors,eachwiththeirownproprietarysoftwareandsearchingalgorithms.In1999,theFBIlaunchedtheIntegratedAutomatedFingerprintIdentificationSystem(IAFIS),anationalfingerprintdatabasethathousesfingerprintsandcriminalhistoriesonmorethan70millionsubjectssubmittedbystate,localandfederallawenforcementagencies(recentlyreplacedbytheNextGenerationIdentification(NGI)System).Somecriminaljusticeagencieshavetheabilitytosearchlatentprintsnotonlyagainsttheirownfingerprintdatabasebutalsoagainstahierarchyoflocal,state,andfederaldatabases.System-wideinteroperability,however,hasyettobeachieved.See:CommitteeonScience,SubcommitteeonForensicScienceoftheNationalScienceandTechnologyCouncil.“AchievingInteroperabilityforLatentFingerprintIdentificationintheUnitedStates.”(2014).www.whitehouse.gov/sites/default/files/microsites/ostp/NSTC/afis_10-20-2014_draftforcomment.pdf.242Thealgorithmsusedingeneratingcandidatematchesareproprietaryandhavenotbeenmadepubliclyavailable.243TheFBILaboratoryrequiresexaminerstocompleteanddocumenttheiranalysisofthelatentfingerprintbeforereviewinganyknownfingerprintsormovingtothecomparisonandevaluationphase,thisthisrequirementisnotsharedbyalllabs.


68

fromtheclosestcandidatematchesgeneratedbythecomputerbystudyingselectedfeatures244andthencomestoasubjectivedecisionastowhethertheyaresimilarenoughtodeclareaproposedidentification.

ACE-Vaddsaverificationstep.Fortheverificationstep,implementationvarieswidely.245Inmanylaboratories,onlyidentificationsareverified,becauseitisconsideredtooburdensome,intermsoftimeandcost,toconductindependentexaminationsinallcases(forexample,exclusions).Thisprocedureisproblematicbecauseitisnotblind:thesecondexaminerknowsthefirstexaminerreachedaconclusionofproposedidentification,whichcreatesthepotentialforconfirmationbias.IntheaftermathoftheMadridtrainbombingcasemisidentification(seebelow),theFBILaboratoryadoptedrequirementstoconduct,incertaincases,“independentapplicationofACEtoafrictionridgeprintbyanotherqualifiedexaminer,whodoesnotknowtheconclusionoftheprimaryexaminer.”246Inparticular,theFBILaboratoryusesblindverificationincasesconsideredtopresentthegreatestriskoferror,suchaswhereasinglefingerprintisidentified,excluded,ordeemedinconclusive.247

AsnotedinChapter2,earlierconcerns248aboutthereliabilityoflatentfingerprintanalysisincreasedsubstantiallyfollowingaprominentmisidentificationofalatentfingerprintrecoveredfromthe2004bombingoftheMadridcommutertrainsystem.AnFBIexaminerconcludedwith“100%certainty”thatthefingerprintmatchedBrandonMayfield,anAmericaninPortland,Oregon,eventhoughSpanishauthoritieswereunabletoconfirmtheidentification.Reviewersbelievethemisidentificationresultedinpartfrom“confirmationbias”and“reversereasoning”—thatis,goingfromtheknownprinttothelatentimageinawaythatledtooverrelianceonapparentsimilaritiesandinadequateattentiontodifferences.249AsdescribedinarecentpaperbyscientistsattheFBILaboratory,

AnotableexampleoftheproblemofbiasfromtheexemplarresultingincircularreasoningoccurredintheMadridmisidentification,inwhichtheinitialexaminerreinterpretedfiveoftheoriginalsevenanalysis

244Fingerprintfeaturesarecomparedatthreelevelsofdetail—level1(“ridgeflow”),level2(“ridgepath”),andlevel3(“ridgefeatures”or“shapes”).”).244“Ridgeflow”referstoclassesofpatterntypessharedbymanyindividuals,suchaslooporwhorlformations;thislevelisonlysufficientforexclusions,notfordeclaringidentifications.“Ridgepath”referstominutiaethatcanbeusedfordeclaringidentifications,suchasbifurcationsordots.“Ridgeshapes”includetheedgesofridgesandlocationofpores.See:NationalInstituteofStandardsandTechnology.“LatentPrintExaminationandHumanFactors:ImprovingthePracticethroughaSystemsApproach.”(2012),availableat:www.nist.gov/oles/upload/latent.pdf.245Black,J.P.“Isthereaneedfor100%verification(review)oflatentprintexaminationconclusions?”JournalofForensicIdentification,Vol.62,No.1(2012):80-100.246U.S.DepartmentofJustice,OfficeoftheInspectorGeneral.“AReviewoftheFBI’sProgressinRespondingtotheRecommendationsintheOfficeoftheInspectorGeneralReportontheFingerprintMisidentificationintheBrandonMayfieldCase.”(2011).www.oig.justice.gov/special/s1105.pdf.Seealso:FederalBureauofInvestigation.LaboratoryDivision.LatentPrintOperationsManual:StandardOperatingProceduresforExaminingFrictionRidgePrints.FBILaboratory,Quantico,Virginia,2007(updatedMay24,2011).247FederalBureauofInvestigation.LaboratoryDivision.LatentPrintOperationsManual:StandardOperatingProceduresforExaminingFrictionRidgePrints.FBILaboratory,Quantico,Virginia,2007(updatedMay24,2011).248Faigman,D.L.,Kaye,D.H.,Saks,M.J.,andJ.Sanders(Eds).ModernScientificEvidence:TheLawandScienceofExpertTestimony,2015-2016ed.Thomson/WestPublishing(2016).Saks,M.J.“ImplicationsofDaubertforforensicidentificationscience.”1Shepard’sExpertandScienceEvidenceQuarterly427,(1994).249AReviewoftheFBI’shandlingoftheBrandonMayfieldCase.U.S.DepartmentofJustice,OfficeoftheInspectorGeneral(2006).oig.justice.special/s0601/final.pdf.


69

pointstobemoreconsistentwiththe(incorrect)exemplar:‘‘Havingfoundasmanyas10pointsofunusualsimilarity,theFBIexaminersbeganto‘find’additionalfeaturesinLFP17[thelatentprint]thatwerenotreallythere,butrathersuggestedtotheexaminersbyfeaturesintheMayfieldprints.’’250

IncontrasttoDNAanalysis,therulesfordeclaringanidentificationthatwerehistoricallyusedinfingerprintanalysiswerenotsetinadvancenoruniformamongexaminers.AsdescribedbyaFebruary2012reportfromanExpertWorkingGroupcommissionedbyNISTandNIJ:

Thethresholdsforthesedecisionscanvaryamongexaminersandamongforensicserviceproviders.Someexaminersstatethattheyreportidentificationiftheyfindaparticularnumberofrelativelyrareconcurringfeatures,forinstance,eightortwelve.Othersdonotuseanyfixednumericalstandard.Someexaminersdiscountseeminglydifferentdetailsaslongasthereareenoughsimilaritiesbetweenthetwoprints.Otherexaminerspracticetheone-dissimilarityrule,excludingaprintifasingledissimilaritynotattributabletoperceptibledistortionexists.Iftheexaminerdecidesthatthedegreeofsimilarityfallsshortofsatisfyingthestandard,theexaminercanreportaninconclusiveoutcome.Iftheconclusionisthatthedegreeofsimilaritysatisfiesthestandard,theexaminerreportsanidentification.251

InSeptember2011,theScientificWorkingGrouponFrictionRidgeAnalysis,StudyandTechnology(SWGFAST)issued“StandardsforExaminingFrictionRidgeImpressionsandResultingConclusions(Latent/Tenprint)”thatbeginstomovelatentprintanalysisinthedirectionofanobjectiveframework.Inparticular,itsuggestscriteriaconcerningwhatcombinationofimagequalityandfeaturequantity(forexample,thenumberof“minutiae”sharedbetweentwofingerprints)wouldbesufficienttodeclareanidentification.Thecriteriaarenotyetfullyobjective,buttheyareastepintherightdirection.TheFrictionRidgeSubcommitteeoftheOSAChasrecognizedtheneedforobjectivecriteriainitsidentificationof“ResearchNeeds.”252Wenotethattheblack-boxstudiesdescribedbelowdidnotsetouttotestthesespecificcriteria,andsotheyhavenotyetbeenscientificallyvalidated.


Asdiscussedabove,thefoundationalvalidityofasubjectivemethodcanonlybeestablishedthroughmultipleindependentblack-boxstudiesappropriatelydesignedtoassessvalidityandreliability.

Below,wediscussvariousstudiesoflatentfingerprintanalysis.Thefirstfivestudieswerenotintendedasvalidationstudies,althoughtheyprovidesomeincidentalinformationaboutperformance.Remarkably,therehavebeenonlytwoblack-boxstudiesthatwereintentionallyandappropriatelydesignedtoassessvalidityand 250Ulery,B.T.,Hicklin,R.A.,Roberts,M.A.,andJ.Buscaglia.“Changesinlatentfingerprintexaminers’markupbetweenanalysisandcomparison.”ForensicScienceInternational,Vol.247(2015):54-61.TheinternalquotationisfromUSDepartmentofJusticeOfficeoftheInspectorGeneral:AreviewoftheFBI'shandlingoftheBrandonMayfieldcase(March2006),http://www.justice.gov/oig/special/s0601/PDF_list.htmUSDepartmentofJusticeOfficeoftheInspectorGeneral:AreviewoftheFBI'shandlingoftheBrandonMayfieldcase(March2006),http://www.justice.gov/oig/special/s0601/PDF_list.htm251See:NIST.“LatentPrintExaminationandHumanFactors:ImprovingthePracticethroughaSystemsApproach.”(2012),availableat:www.nist.gov/oles/upload/latent.pdf.252See:workspace.forensicosac.org/kws/groups/fric_ridge/documents.


70

reliability—thefirstpublishedbytheFBILaboratoryin2011;thesecondcompletedin2014butnotyetpublished.Conclusionsaboutfoundationalvaliditythusmustrestonthesetworecentstudies.

Insummarizingthesestudies,weapplytheguidelinesdescribedearlierinthisreport(seeChapter4andAppendixA).First,whilewenote(1)boththeestimatedfalsepositiveratesand(2)theupper95%confidenceboundonthefalsepositiverate,wefocusonthelatteras,fromascientificperspective,theappropriateratetoreporttoajury—becausetheprimaryconcernshouldbeaboutunderestimatingthefalsepositiverateandthetrueratecouldreasonablybeashighasthisvalue.253Second,whilewenoteboththefalsepositiverateamongconclusiveexaminations(identificationsorexclusions)oramongallexaminations(includinginconclusives)arerelevant,wefocusprimarilyontheformerasbeing,fromascientificperspective,theappropriateratetoreporttoajury—becausefingerprintevidenceusedagainstadefendantincourtwilltypicallybetheresultofaconclusiveexamination.

EvettandWilliams(1996)Thispaperisadiscursivehistoricalreviewessaythatcontainsabriefdescriptionofasmall“collaborativestudy”relevanttotheaccuracyoffingerprintanalysis.254Inthisstudy,130highlyexperiencedexaminersinEnglandandWales,eachwithatleasttenyearsofexperienceinforensicfingerprintanalysis,werepresentedwithtenlatentprint-knownpairs.NineofthepairscamefrompastcaseworkatNewScotlandYardandwerepresumedtobe‘matedpairs’(thatis,fromthesamesource).Thetenthpairwasa‘non-matedpair’(fromdifferentsources),involvingalatentprintdeliberatelyproducedona“dimpledbeermug.”Forthesinglenon-matedpair,the130expertsmadenofalseidentifications.Becausethepaperdoesnotdistinguishbetweenexclusionsandinconclusiveexaminations(andtheauthorsnolongerhavethedata255),itisimpossibletoinfertheupper95%confidencebound.256

Langenburg(2009a)Inasmallpilotstudy,theauthorexaminedtheperformanceofsixexaminerson60testseach.257Therewereonly15conclusiveexaminationsinvolvingnon-matedpairs(seeTable1ofthepaper).Therewasonefalsepositive,whichtheauthorexcludedbecauseitappearedtobeaclericalerrorandwasnotrepeatedonsubsequentretest.Evenifthiserrorisexcluded,thetinysamplesizeresultsinahugeconfidenceinterval(upper95%confidenceboundof19%),withthisupperboundcorrespondingto1errorin5cases.

253Byconvention,the95%confidenceboundismostwidelyusedinstatisticsasreflectingtherangeofplausiblevalues(seeAppendixA).254Evett,I.W.,andR.L.Williams.“Reviewofthe16pointfingerprintstandardinEnglandandWales.”ForensicScienceInternational,Vol.46,No.1(1996):49-–73.255I.W.Evett,personalcommunication.256Forexample,theupper95%confidenceboundwouldbe1in44ifall130examinationswereconclusiveand1in22ifhalfoftheexaminationswereconclusive.257Langenburg,G.“AperformancestudyoftheACE-VProcess:Apilotstudytomeasuretheaccuracy,precision,reproducibility,repeatability,andbiasabilityofconclusionsresultingfromtheACE-Vprocess.”JournalofForensicIdentification,Vol.59,No.2(2009):219–57.


71

Langenburg(2009b)Inthissmallpilotstudyforthefollowingpaper,theauthortestedexaminersinaconferenceroomataconventionofforensicidentificationspecialists.258Theexaminersweredividedintothreegroups:high-bias(n=16),low-bias(n=12),andcontrol(n=15).Eachgroupwaspresentedwith6latent-knownpairs,consistingof3matedand3non-matedpairs.Thefirsttwogroupsreceivedinformationdesignedtobiastheirjudgmentbyheighteningtheirattention,whilethecontrolgroupreceivedagenericdescription.Forthenon-matedpairs,thecontrolgrouphad1falsepositiveamong43conclusiveexaminations.Thefalsepositiveratewas2.3%(upper95%confidenceboundof11%),withtheupperboundcorrespondingto1errorin9cases.259,260

Langenburg,Champod,andGenessay(2012)Thisstudywasnotdesignedtoassesstheaccuracyoflatentfingerprintanalysis,butrathertoexplorehowfingerprintanalystswouldincorporateinformationfromnewlydevelopedtools(suchasaqualitytooltoaidintheassessmentoftheclarityofthefrictionridgedetails;astatisticaltooltoprovidelikelihoodratiosrepresentingthestrengthofthecorrespondingfeaturesbetweencomparedfingerprints;andconsensusinformationfromagroupoftrainedfingerprintexperts)intotheirdecisionmakingprocesses.261Nonetheless,thestudyprovidedsomeinformationontheaccuracyoflatentprintanalysis.Briefly,158experts(aswellassometrainees)wereaskedtoanalyze12latentprint-exemplarpairs,consistingof7matedand5non-matedpairs.Forthenon-matedpairs,therewere17falsepositivematchesamong711conclusiveexaminationsbytheexperts.262Thefalsepositiveratewas2.4%(upper95%confidenceboundof3.5%).Theestimatederrorratecorrespondsto1errorin42cases,withanupperboundcorrespondingto1errorin28cases.263

Tangenetal.(2011)ThisAustralianstudywasdesignedtostudythereliabilityoflatentfingerprintanalysisbyfingerprintexperts.264Theauthorsasked37fingerprintexperts,aswellas37novices,toexamine36latentprint-knownpairs—consistingof12matedpairs,12non-matedpairschosentobe“similar”(themosthighlyrankedexemplarfromadifferentsourceintheAustralianNationalAutomatedFingerprintIdentificationSystem),and12“non-similar”non-matedpairs(chosenatrandomfromtheotherprints).Examinerswereaskedtoratethelikelihoodtheycamefromthesamesourceonascalefrom1to12.Theauthorschosetodefinescoresof1-6asidentifications

258Langenburg,G.,Champod,C.,andP.Wertheim.“TestingforpotentialcontextualbiaseffectsduringtheverificationstageoftheACE-Vmethodologywhenconductingfingerprintcomparisons.”JournalofForensicSciences,Vol.54,No.3(2009):571-82.259Ifthetwoinconclusiveexaminationsareincluded,thevaluesareonlyslightlydifferent:2.2%(upper95%confidenceboundof10.1%),withtheoddsbeing1in10.260Thebiasedgroupsmadenoerrorsamong69conclusiveexaminations.261Langenburg,G.,Champod,C.,andT.Genessay.“Informingthejudgmentsoffingerprintanalystsusingqualitymetricandstatisticalassessmenttools.”ForensicScienceInternational,Vol.219,No.1-3(2012):183-98.262WethankG.Langenburgforprovidingthedatafortheexpertsalone.263Ifthe79inconclusiveexaminationsareincluded,thefalsepositiveratewas2.15%(upper95%confidenceboundof3.2%).Theestimatedfalsepositiveratecorrespondsto1errorin47cases,withtheupperboundcorrespondingto1in31.264Tangen,J.M.,Thompson,M.B.,andD.J.McCarthy.“Identifyingfingerprintexpertise.”PsychologicalScience,Vol.22,No.8(2011):995-7.


72

andscoresof7-12asexclusions.265Thisapproachdoesnotcorrespondtotheproceduresusedinconventionalfingerprintexamination.

Forthe“similar”non-matedpairs,theexpertsmade3errorsamong444comparisons;thefalsepositiveratewas0.68%(upper95%confidenceboundof1.7%),withtheupperboundcorrespondingto1errorin58cases.Forthe“non-similar”non-matedpairs,theexaminersmadenoerrorsin444comparisons;thefalsepositiveratewasthus0%(upper95%confidenceboundof0.62%),withtheupperboundcorrespondingto1errorin148cases.Theexpertssubstantiallyoutperformedthenovices.

Althoughinteresting,thestudydoesnotconstituteablack-boxvalidationstudyoflatentfingerprintanalysisbecauseitsdesigndidnotresembletheproceduresusedinforensicpractice(inparticular,theprocessofassigningratingona12-pointscalethattheauthorssubsequentlyconvertedintoidentificationsandexclusions).

FBIstudiesThefirststudydesignedtotestfoundationalvalidityandmeasurereliabilityoflatentfingerprintanalysiswasamajorblack-boxstudyconductedbyFBIscientistsandcollaborators.Undertakeninresponsetothe2009NRCreport,thestudywaspublishedin2011inaleadinginternationalsciencejournal,ProceedingsoftheNationalAcademyofSciences.266Theauthorsassembledacollectionof744latent-knownpairs,consistingof520matedpairsand224non-matedpairs.Toattempttoensurethatthenon-matedpairswererepresentativeofthetypeofmatchesthatmightarisewhenpoliceidentifyasuspectbysearchingfingerprintdatabases,theknownprintswereselectedbysearchingthelatentprintsagainstthe58millionfingerprintsintheAFISdatabaseandselectingoneoftheclosestmatchinghits.Eachof169fingerprintexaminerswasshown100pairsandaskedtoclassifythemasanidentification,anexclusion,orinconclusive.Thestudyreported6falsepositiveidentificationsamong3628nonmatedpairsthatexaminersjudgedtohave“valueforidentification.”Thefalsepositiveratewasthus0.17%(upper95%confidenceboundof0.33%).Theestimatedratecorrespondsto1errorin604cases,withtheupperboundindicatingthattheratecouldbeashighas1errorin306cases.267,268

In2012,thesameauthorsreportedafollow-upstudytestingrepeatabilityandreproducibility.Afteraperiodofaboutsevenmonths,75oftheexaminersfromthepreviousstudyre-examinedasubsetofthelatent-knowncomparisonsfromthepreviousstudy.Among476nonmatedpairsleadingtoconclusiveexaminations(including4ofthepairsthatledtofalsepositivesintheinitialstudyandwerereassignedtotheexaminerwhohadmadetheerroneousdecision),therewerenofalsepositives.Theseresults(upper95%confidenceboundof0.63%,

265Therewerethusnoinconclusiveresultsinthisstudy.266Ulery,B.T.,Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Accuracyandreliabilityofforensiclatentfingerprintdecisions.”ProceedingsoftheNationalAcademyofSciences,Vol.108,No.19(2011):7733-8.267Ifoneincludesthe455inconclusiveresultsforlatentprintsjudgedtohave“valueforidentification,”thefalsepositiverateis0.15%(upper95%confidenceboundof0of0.29%).Theestimatedfalsepositiveratecorrespondsto1errorin681cases,withtheupperboundcorrespondingto1in344.268Thesensitivity(proportionofmatedsamplesthatwerecorrectlydeclaredtomatch)was92.5%.


73

correspondingto1errorin160)arebroadlyconsistentwiththefalsepositiveratemeasuredinthepreviousstudy.269

Miami-Dadestudy(Pachecoetal.(2014))TheMiami-DadePoliceDepartmentForensicServicesBureau,withfundingfromtheNIJ,conductedablack-boxstudydesignedtoassessfoundationalvalidityandmeasurereliability;theresultswerereportedtothesponsorandpostedontheinternet,buttheyhavenotyetpublishedinapeer-reviewedscientificjournal.270Thestudydifferedsignificantlyfromthe2011FBIblack-boxstudyinimportantrespects,includingthattheknownprintswerenotselectedbymeansofalargedatabasesearchtobesimilartothelatentprints(whichshould,inprinciple,havemadeiteasiertodeclareexclusionsforthenon-matedpairs).Thestudyfound42falsepositivesamong995conclusiveexaminations.Thefalsepositiveratewas4.2%(upper95%confidenceboundof5.4%).Theestimatedratecorrespondsto1errorin24cases,withtheupperboundindicatingthattheratecouldbeashighas1errorin18cases.271(Note:Thepaperobservesthat“in35oftheerroneousidentificationstheparticipantsappearedtohavemadeaclericalerror,buttheauthorscouldnotdeterminethiswithcertainty.”Invalidationstudies,itisinappropriatetoexcludeerrorsinaposthocmanner(seeBox4).However,ifthese35errorsweretobeexcluded,thefalsepositiveratewouldbe0.7%(confidenceinterval1.4%),withtheupperboundcorrespondingto1errorin73cases.)

Conclusionsfromthestudies

Whileitisdistressingthatmeaningfulstudiestoassessfoundationalvalidityandreliabilitydidnotbeginuntilrecently,weareencouragedthatseriouseffortsarenowbeingmadetotrytoputthefieldonasolidscientificfoundation—includingbymeasuringaccuracy,definingqualityoflatentprints,studyingthereasonforerrors,andsoon.MuchcreditbelongstotheFBILaboratory,aswellastoacademicresearcherswhohadbeenpressingtheneedforresearch.Importantly,theFBIisresponsiblefortheonlyblack-boxstudytodatethathasbeenpublishedinapeer-reviewedjournal.

Thestudiesabovecannotbedirectlycomparedformanyreasons—includingdifferencesinexperimentaldesign,selectionanddifficultyleveloflatent-knownpairs,anddegreetowhichtheyrepresentthecircumstances,proceduresandpressuresfoundincasework.Nonetheless,certainconclusionscanbedrawnfromtheresultsofthestudies(summarizedinTable1below):

(1)Thestudiescollectivelydemonstratethatmanyexaminerscan,undersomecircumstances,producecorrectanswersatsomelevelofaccuracy.

269Overall,85-90%oftheconclusiveresultswereunchanged,withroughly30%offalseexclusionsbeingrepeated.270Pacheco,I.,Cerchiai,B.,andS.Stoiloff.“Miami-DaderesearchstudyforthereliabilityoftheACE-Vprocess:Accuracy&precisioninlatentfingerprintexaminations.”(2014).www.ncjrs.gov/pdffiles1/nij/grants/248534.pdf.271Ifthe403inconclusiveexaminationsareincluded,thefalsepositiveratewas3.0%(upper95%confidenceboundof3.9%).Theestimatedfalsepositiveratecorrespondsto1errorin33cases,withtheupperboundcorrespondingto1in26.


74

(2)Theempiricallyestimatedfalsepositiveratesaremuchhigherthanthegeneralpublic(and,byextension,mostjurors)wouldlikelybelievebasedonlongstandingclaimsabouttheaccuracyoffingerprintanalysis.272,273

(3)Ofthetwoappropriatelydesignedblack-boxstudies,thelargerstudy(FBI2011study)yieldedafalsepositiveratethatisunlikelytoexceed1in306conclusiveexaminationswhiletheother(Miami-Dade2014study)yieldedconsiderablyhigherfalsepositiverateof1in18.274(Theearlierstudies,whichwerenotdesignedasvalidationstudies,alsoyieldedhighfalsepositiverates.)

Overall,itwouldbeappropriatetoinformjurorsthat(1)onlytwoproperlydesignedstudiesoftheaccuracyoflatentfingerprintanalysishavebeenconductedand(2)thesestudiesfoundfalsepositiveratesthatcouldbeashighas1in306inonestudyand1in18intheotherstudy.Thiswouldappropriatelyinformjurorsthaterrorsoccuratdetectablefrequencies,allowingthemtoweightheprobativevalueoftheevidence.

Itislikelythataproperlydesignedprogramofsystematic,blindverificationwoulddecreasethefalse-positiverate,becauseexaminersinthestudiestendtomakedifferentmistakes.275However,therehasnotbeenempiricaltestingtoobtainaquantitativeestimateofthefalsepositiveratethatmightbeachievedthroughsuchaprogram.276And,itwouldnotbeappropriatesimplytoinfertheimpactofindependentverificationbasedonthetheoreticalassumptionthatexaminers’errorsareuncorrelated.277

272Theconclusionholdsregardlessofwhethertheratesarebasedonthepointestimatesorthe95%confidencebound,andonconclusiveexaminationsorallexaminations.273TheseclaimsincludetheDOJ’sownlongstandingpreviousassertionthatfingerprintanalysisis“infallible”(https://www.justice.gov/olp/file/861906/download.);testimonybyaformerheadoftheFBI’sfingerprintunittestifiedthattheFBIhad“anerrorrateofoneperevery11millioncases”(seep.26);andastudyfindingthatmockjurorsestimatedthatthefalsepositiverateforlatentfingerprintanalysisis1in5.5million(seep.26).JonathanJ.Koehler.“Intuitiveerrorrateestimatesfortheforensicsciences.”(August2,2016).AvailableatSSRN:http://ssrn.com/abstract=2817443orhttp://dx.doi.org/10.2139/ssrn.2817443]274Asnotedabove,therateis1in73ifoneignoresthepresumedclericalerrors—althoughsuchposthocadjustmentisnotappropriateinvalidationstudies.275TheauthorsoftheFBIblack-boxstudynotethatfiveofthefalsepositiveoccurredontestproblemwherealargemajorityofexaminerscorrectlydeclaredanexclusion,whileoneoccurredonatestproblemwherethemajorityofexaminersmadeinconclusivedecisions.Theystatethat“thissuggeststhattheseerroneousindividualizationswouldhavebeendetectedifblindverificationwereroutinelyperformed.”Ulery,B.T.,Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Accuracyandreliabilityofforensiclatentfingerprintdecisions.”ProceedingsoftheNationalAcademyofSciences,Vol.108,No.19(2011):7733-8.276TheMiami-Dadestudyinvolvedasmalltestofverificationstep,involvingverificationof15ofthe42falsepositives.Inthese15cases,thesecondexaminerdeclared13casestobeexclusionsand2tobeinconclusive.Thesamplesizeistoosmalltodrawameaningfulconclusion.And,thepaperdoesnotreportverificationresultsfortheother27falsepositives.277TheDOJhasproposedtoPCASTthat“basicprobabilitystatesthatgivenanerrorrateforoneexaminer,thelikelihoodofasecondexaminermakingtheexactsameerror(verification/blindverification),woulddictatethattheratesshouldbemultiplied.”However,suchatheoreticalmodelwouldassumethaterrorsbydifferentexaminerswillbeuncorrelated;yettheymaydependonthedifficultyoftheproblemandthusbecorrelated.Empiricalstudiesarenecessarytoestimateerrorratesunderblindverification.


75

Itisimportanttonotethat,foraverificationprogramtobetrulyblindandtherebyavoidcognitivebias,examinerscannotonlyverifyindividualizations.AstheauthorsoftheFBIblack-boxstudypropose,“thiscanbeensuredbyperformingverificationsonamixofconclusiontypes,notmerelyindividualizations”—thatis,amixthatensuresthatverifierscannotmakeinferencesabouttheconclusionsbeingverified.278Wearenotawareofanyblindverificationprogramsthatcurrentlyfollowthispractice.

Atpresent,testimonyassertinganyspecificlevelofincreasedaccuracy(beyondthatmeasuredinthestudies)duetoblindindependentverificationwouldbescientificallyinappropriate,asspeculationunsupportedbyempiricalevidence.

WenotethattheDOJbelievesthatthehighfalsepositiverateobservedintheMiami-Dadestudy(1in24,withupperconfidencelimitof1in18)isunlikelytoapplytocaseworkattheFBILaboratory,becauseitbelievessuchahighratewouldhavebeendetectedbytheLaboratory’sverificationprocedures.AnindependentevaluationoftheverificationprotocolscouldshedlightontheextenttowhichsuchinferencescouldbedrawnbasedonthecurrentLaboratory’sverificationprocedures.

Wealsonoteitisconceivablethatthefalse-positiverateinrealcaseworkcouldbehigherthanthatobservedintheexperimentalstudies,duetoexposuretopotentiallybiasinginformationinthecourseofcasework.Introducingtestsamplesblindlyintotheflowofcaseworkcouldprovidevaluableinsightabouttheactualerrorratesincasework.

Inconclusion,theFBIblack-boxstudyhassignificantlyadvancedthefield.Thereisaneedforongoingstudiesofthereliabilityoflatentprintanalysis,buildingonitsstudydesign.Studiesshouldideallyestimateerrorratesforlatentprintsofvarying“quality”levels,usingwelldefinedmeasures(ideally,objectivemeasuresimplementedbyautomatedsoftware279).Asnotedabove,studiesshouldbedesignedandconductedinconjunctionwiththirdpartieswithnostakeintheoutcome.ThisimportantfeaturewasnotpresentintheFBIstudy.

278Ulery,B.T.,Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Accuracyandreliabilityofforensiclatentfingerprintdecisions.”ProceedingsoftheNationalAcademyofSciences,Vol.108,No.19(2011):7733-8.279AnexampleistheLatentQualityAssessment(LQAS),whichisdesignedasaproof-of-concepttooltoevaluatetheclarityofprints.Studieshavefoundthaterrorratesarecorrelatedtothequalityoftheprint.Thesoftwareprovidesamanualandautomateddefinitionsofclaritymaps,functionstoprocessclaritymaps,andannotationofcorrespondingpointsprovidingamethodforoverlappingofimpressionareas.Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Assessingtheclarityoffrictionridgeimpressions.”ForensicScienceInternational,Vol.226,No.1(2013):106-17.AnotherexampleisthePictureAnnotationSystem(PiAnoS),developedbytheUniversityofLausanne,whichisbeingtestedasaqualitymetricandstatisticalassessmenttoolforanalysts.Thisplatformusestoolsthat(1)assesstheclarityofthefrictionridgedetails,(2)providelikelihoodratiosrepresentingthestrengthofcorrespondingfeaturesbetweenfingerprints,and(3)givesconsensusinformationfromagroupoftrainedfingerprintexperts.PiAnoSisanopen-sourcesoftwarepackageavailableat:ips-labs.unil.ch/pianos.


76

Table1:Errorratesinstudiesoflatentprintanalysis*Study FalsePositives

RawData

Freq.(Confidencebound)

EstimatedRate

BoundonRate

Earlystudies

Langenburg(2009a) 0/14 0%(19%) 1in∞ 1in5

Langenburg(2009b) 1/43 2.3%(11%) 1in43 1in9

Langenburgetal.(2012) 17/711 2.4%(3.5%) 1in42 1in28

Tangenetal.(2011)(“similarpairs”) 3/444 0.68%(1.7%) 1in148 1in58

Tangenetal.(2011)(“dissimilarpairs”) 0/444 0%(0.67%) 1in∞ 1in148

Black-boxstudies

Uleryetal.2011(FBI)** 6/3628 0.17%(0.33%) 1in604 1in306

Pachecoetal.2014(Miami-Dade) 42/995 4.2%(5.4%) 1in24 1in18

Pachecoetal.2014(Miami-Dade)(excludingclericalerrors)

7/960 0.7%(1.4%) 1in137 1in73

*“RawData”:Numberoffalsepositivesdividedbynumberofconclusiveexaminationsinvolvingnon-matedpairs.“Freq.(ConfidenceBound)”:Pointestimateoffalsepositivefrequency,andupper95%confidencebound.“EstimatedRate”:Theoddsofafalsepositiveoccurring,basedontheobservedproportionoffalsepositives.“BoundonRate”:Theoddsofafalsepositiveoccurring,basedontheupper95%confidencebound—thatis,theratecouldreasonablybeashighasthisvalue.**IfinconclusiveexaminationsareincludedfortheFBIstudy,theratesare1in681and1in344,respectively.

Scientificstudiesofhowlatent-printexaminersreachconclusions

Complementingtheblack-boxstudies,variousstudieshaveshedimportantlightonhowlatentfingerprintexaminersreachconclusionsandhowtheseconclusionsmaybeinfluencedbyextraneousfactors.Thesestudiesunderscoretheseriousrisksthatmayariseinsubjectivemethods.

Cognitive-biasstudiesItielDrorandcolleagueshavedonepioneeringworkonthepotentialroleofcognitivebiasinlatentfingerprintanalysis.280Inanexploratorystudyin2006,theydemonstratedthatexaminers’judgmentscanbeinfluencedbyknowledgeaboutotherforensicexaminers’decisions(aformof“confirmationbias”).281Fivefingerprintexaminersweregivenfingerprintpairsthattheyhadstudiedfiveyearsearlierinrealcasesandhadjudgedto“match.”Theywereaskedtore-examinetheprints,butwereledtobelievethattheywerethepairofprintsthathadbeenerroneouslymatchedbytheFBIinahigh-profilecase.Althoughtheywereinstructedtoignorethisinformation,fouroutoffiveexaminersnolongerjudgedtheprintsto“match.”Althoughthesestudiesare

280Dror,I.E.,Charlton,D.,andA.E.Peron.“Contextualinformationrendersexpertsvulnerabletomakingerroneousidentifications.”ForensicScienceInternational,Vol.156(2006):74-878.Dror,I.E.,andD.Charlton.“Whyexpertsmakeerrors.”JournalofForensicidentification,Vol.56,No.4(2006):600-16616.281Dror,I.E.,Charlton,D.,andA.E.Peron.“Contextualinformationrendersexpertsvulnerabletomakingerroneousidentifications.”ForensicScienceInternational,Vol.156(2006):74-878.


77

toosmalltoprovidepreciseestimatesoftheimpactofcognitivebias,theyhavebeeninstrumentalincallingattentiontotheissue.

Severalstrategieshavebeenproposedformitigatingcognitivebiasinforensiclaboratories,includingmanagingtheflowofinformationinacrimelaboratorytominimizeexposureoftheforensicanalysttoirrelevantcontextualinformation(suchasconfessionsoreyewitnessidentification)andensuringthatexaminersworkinalinearfashion,documentingtheirfindingaboutevidencefromcrimesciencebeforeperformingcomparisonswithsamplesfromasuspect.282,283

FBIwhite-boxstudiesInthepastfewyears,FBIscientistsandtheircollaboratorshavealsoundertakenaseriesof“white-box”studiestounderstandthefactorsunderlyingtheprocessoflatentfingerprintanalysis.Thesestudiesincludeanalysesoffingerprintquality,284,285examiners’processestodeterminethevalueofalatentprintforidentificationorexclusion,286thesufficiencyofinformationforidentifications,287andhowexaminers’assessmentsofalatentprintchangewhentheycompareitwithapossiblematch.288

Amongworkonsubjectivefeature-comparisonmethods,thisseriesofpapersisuniqueinitsbreadth,rigorandwillingnesstoexplorechallengingissues.Wecouldfindnosimilarlyself-reflectiveanalysesforothersubjectivedisciplines.

Thetwomostrecentpapersareparticularlynotablebecausetheyinvolvetheseriousissueofconfirmationbias.Ina2014paper,theFBIscientistswrote

ACEdistinguishesbetweentheComparisonphase(assessmentoffeatures)andEvaluationphase(determination),implyingthatdeterminationsarebasedontheassessmentoffeatures.However,ourresultssuggestthatthisisnotasimplecausalrelation:examiners’markupsarealsoinfluencedbytheirdeterminations.Howthisreverseinfluenceoccursisnotobvious.Examinersmaysubconsciouslyreacha

282Kassin,S.M.,Dror,I.E.,andJ.Kakucka.“Theforensicconfirmationbias:Problems,perspectives,andproposedsolutions.”JournalofAppliedResearchinMemoryandCognition,Vol.2,No.1(2013):42-52.Seealso:Krane,D.E.,Ford,S.,Gilder,J.,Iman,K.,Jamieson,A.,Taylor,M.S.,andW.C.Thompson.“Sequentialunmasking:AmeansofminimizingobservereffectsinforensicDNAinterpretation.”JournalofForensicSciences,Vol.53,No.4(July2008):1006-7.283Irrelevantcontextualinformationcould,dependingonitsnature,biasanexaminertowardanincorrectidentificationoranincorrectexclusion.Eitheroutcomeisundesirable.284Hicklin,R.A.,Buscaglia,J.,Roberts,M.A.,Meagher,S.B.,Fellner,W.,Burge,M.J.,Monaco,M.,Vera,D.,Pantzer,L.R.,Yeung,C.C.,andN.Unnikumaran.“Latentfingerprintquality:asurveyofexaminers.”JournalofForensicIdentification.Vol.61,No.4(2011):385-419.285Hicklin,R.A.,Buscaglia,J.,andM.A.Roberts.“Assessingtheclarityoffrictionridgeimpressions.”ForensicScienceInternational,Vol.226,No.1(2013):106-17.286Ulery,B.T.,Hicklin,R.A.,Kiebuzinski,G.I.,Roberts,M.A.,andJ.Buscaglia.“Understandingthesufficiencyofinformationforlatentfingerprintvaluedeterminations.”ForensicScienceInternational,Vol.230,No.1-3(2013):99-106.287Ulery,B.T.,Hicklin,R.A.,andJ.Buscaglia.“Repeatabilityandreproducibilityofdecisionsbylatentfingerprintexaminers.”PLoSONE,(2012).288Ulery,B.T.,Hicklin,R.A.,Roberts,M.A.,andJ.Buscaglia.“Changesinlatentfingerprintexaminers’markupbetweenanalysisandcomparison.”ForensicScienceInternational,Vol.247(2015):54-61.


78

preliminarydeterminationquicklyandthisinfluencestheirbehaviorduringComparison(e.g.,levelofeffortexpended,howtotreatambiguousfeatures).Aftermakingadecision,examinersmaythenrevisetheirannotationstohelpdocumentthatdecision,andexaminersmaybemoremotivatedtoprovidethoroughandcarefulmarkupinsupportofindividualizationsthanotherdeterminations.Asevidenceinsupportofourconjecture,wenoteinparticularthedistributionsofminutiacounts,whichshowastepincreaseassociatedwithdecisionthresholds:thisstepoccurredataboutsevenminutiaeformostexaminers,butat12forthoseexaminersfollowinga12-pointstandard.289

SimilarobservationshadbeenmadebyDroretal.,290whonotedthatthenumberofminutiaemarkedinalatentprintwasgreaterwhenamatchingexemplarwaspresent.Inaddition,EvettandWilliamsdescribedhowBritishexaminers,whouseda16-pointstandardfordeclaringidentifications,usedanexemplarto‘‘teasethepointsout’’ofthelatentprintaftertheyhadreachedan‘‘innerconviction’’thattheprintsmatched.291

Inafollow-uppaperin2015,theFBIscientistscarefullystudiedhowexaminersanalyzedprintsandconfirmedthat,inthevastmajority(>90%)ofidentificationdecisions,examinersmodifiedthefeaturesmarkedinthelatentfingerprintinresponsetoanapparentlymatchingknownfingerprint(moreoftenaddingthansubtractingfeatures).292(Thesolefalsepositiveintheirstudywasanextremecaseinwhichtheconclusionwasbasedalmostentirelyonsubsequentmarkingofminutiaethathadnotbeeninitiallyfoundanddeletionoffeaturesthathadbeeninitiallymarked.)

Theauthorsconcludedthat“thereisaneedforexaminerstohavesomemeansofunambiguouslydocumentingwhattheyseeduringanalysisandcomparison(intheACE-Vprocess)”andthat“rigorouslydefinedandconsistentlyappliedmethodsofperforminganddocumentingACE-Vwouldimprovethetransparencyofthelatentprintexaminationprocess.”

PCASTcomplimentstheFBIscientistsforcallingattentiontotheriskofconfirmationbiasarisingfromcircularreasoning.Asamatterofscientificvalidity,examinersmustberequiredto“completeanddocumenttheiranalysisofalatentfingerprintbeforelookingatanyknownfingerprint”and“mustseparatelydocumentanydatarelieduponduringcomparisonorevaluationthatdiffersfromtheinformationrelieduponduring

289Ulery,B.T.,Hicklin,R.A.,Roberts,M.A.,andJ.Buscaglia.“Measuringwhatlatentfingerprintexaminersconsidersufficientinformationforindividualizationdeterminations.”PLoSONE,(2014).290Dror,I.E.,Champod,C.,Langenburg,G.,Charlton,D.,Hunt,H.,andR.Rosenthal.“Cognitiveissuesinfingerprintanalysis:Inter-andintra-expertconsistencyandtheeffectofa‘target’comparison.”ForensicScienceInternational,Vol.208,No.1-3(2011):10-7.291Evett,I.W.,andR.L.Williams.“Reviewofthe16pointfingerprintstandardinEnglandandWales.”ForensicScienceInternational,Vol.46,No.1(1996):49–73.292Ulery,B.T.,Hicklin,R.A.,Roberts,M.A.,andJ.Buscaglia.“Changesinlatentfingerprintexaminers’markupbetweenanalysisandcomparison.”ForensicScienceInternational,Vol.247(2015):54-61.


79

analysis.”293TheFBIadoptedtheserulesfollowingtheMadridtrainbombingcasemisidentification;theyneedtobeuniversallyadoptedbyalllaboratories.

Validityasapplied

Foundationalvaliditymeansthatalargegroupofexaminersanalyzingaspecifictypeofsamplecan,undertestconditions,producecorrectanswersataknownandusefulfrequency.Itdoesnotmeanthataparticularexaminerhastheabilitytoreliablyapplythemethod;thatthesamplesinthefoundationalstudiesarerepresentativeoftheactualevidenceofthecase;orthatthecircumstancesofthefoundationalstudyrepresentareasonableapproximationofthecircumstancesofcasework.

Toaddressthesematters,courtsshouldtakeintoaccountseveralkeyconsiderations.

(1)Becauselatentprintanalysis,ascurrentlypracticed,dependsonsubjectivejudgment,itisscientificallyunjustifiedtoconcludethataparticularexamineriscapableofreliablyapplyingthemethodunlesstheexaminerhasundergoneregularandrigorousproficiencytesting.Unfortunately,itisnotpossibletoassesstheappropriatenessofcurrentproficiencytestingbecausethetestproblemsarenotpublicallyreleased.(Asemphasizedpreviously,trainingandexperiencearenosubstitute,becauseneitherprovidesanyassurancethattheexaminercanapplythemethodreliably.)

(2)Inanygivencase,itmustbeestablishedthatthelatentprint(s)areofthequalityandcompletenessrepresentedinthefoundationalvaliditystudies.

(3)Becausecontextualbiasmayhaveanimpactonexperts’decisions,courtsshouldassessthemeasurestakentomitigatebiasduringcasework—forexample,ensuringthatexaminersarenotexposedtopotentiallybiasinginformationandensuringthatanalystsdocumentridgefeaturesofanunknownprintbeforereferringtotheknownprint(aprocedureknownas“linearACE-V”294).

Finding5:Latentfingerprintanalysis

Foundationalvalidity.Basedlargelyonatworecentappropriatelydesignedblack-boxstudies,PCASTfindsthatlatentfingerprintanalysisisafoundationallyvalidsubjectivemethodology—albeitwithafalsepositiveratethatissubstantialandislikelytobehigherthanexpectedbymanyjurorsbasedonlongstandingclaimsabouttheinfallibilityoffingerprintanalysis.

Conclusionsofaproposedidentificationmaybescientificallyvalid,providedthattheyareaccompaniedbyaccurateinformationaboutlimitationsonthereliabilityoftheconclusion—specifically,that(1)only

293U.S.DepartmentofJustice,OfficeoftheInspectorGeneral.“AReviewoftheFBI’sProgressinRespondingtotheRecommendationsintheOfficeoftheInspectorGeneralReportontheFingerprintMisidentificationintheBrandonMayfieldCase.”(2011).P.5,27.www.oig.justice.gov/special/s1105.pdf.294U.S.DepartmentofJustice,OfficeoftheInspectorGeneral.“AReviewoftheFBI’sProgressinRespondingtotheRecommendationsintheOfficeoftheInspectorGeneralReportontheFingerprintMisidentificationintheBrandonMayfieldCase.”(2011):p.27.www.oig.justice.gov/special/s1105.pdf.


80

twoproperlydesignedstudiesofthefoundationalvalidityandaccuracyoflatentfingerprintanalysishavebeenconducted,(2)thesestudiesfoundfalsepositiveratesthatcouldbeashighas1errorin306casesinonestudyand1errorin18casesintheother,and(3)becausetheexaminerswereawaretheywerebeingtested,theactualfalsepositiverateincaseworkmaybehigher.Atpresent,claimsofhigheraccuracyarenotwarrantedorscientificallyjustified.Additionalblack-boxstudiesareneededtoclarifythereliabilityofthemethod.

Validityasapplied.Althoughweconcludethatthemethodisfoundationallyvalid,thereareanumberofopenissuesrelatedtoitsvalidityasapplied.

(1)Confirmationbias.WorkbyFBIscientistshasshownthatexaminerstypicallyalterthefeaturesthattheyinitiallymarkinalatentprintbasedoncomparisonwithanapparentlymatchingexemplar.Suchcircularreasoningintroducesaseriousriskofconfirmationbias.Examinersshouldberequiredtocompleteanddocumenttheiranalysisofalatentfingerprintbeforelookingatanyknownfingerprintandshouldseparatelydocumentanyadditionaldatausedduringtheircomparisonandevaluation.

(2)Contextualbias.Workbyacademicscholarshasshownthatexaminers’judgmentscanbeinfluencedbyirrelevantinformationaboutthefactsofacase.Effortsshouldbemadetoensurethatexaminersarenotexposedtopotentiallybiasinginformation.

(3)Proficiencytesting.Proficiencytestingisessentialfordetermininganexaminer’sabilitytomakeaccuratejudgments.Asdiscussedelsewhereinthisreport,proficiencytestingneedstobeimprovedbymakingitmorerigorous,byincorporatingitwithintheflowofcaseworkmorerigorous,andbydisclosingtestsforevaluationbythescientificcommunity.

Fromascientificstandpoint,validityasappliedrequiresthatanexpert:(1)hasundergonerelevantproficiencytestingtotesthisorheraccuracyandreportstheresultsoftheproficiencytesting;(2)discloseswhetherheorshedocumentedthefeaturesinthelatentprintinwritingbeforecomparingittotheknownprint;(3)providesawrittenanalysisexplainingtheselectionandcomparisonofthefeatures;(4)discloseswhether,whenperformingtheexamination,heorshewasawareofanyotherfactsofthecasethatmightinfluencetheconclusion;and(5)verifiesthatthelatentprintinthecaseathandissimilarinqualitytotherangeoflatentprintsconsideredinthefoundationalstudies.

ThePathForward

Continuingeffortsareneededtoimprovethestateoflatentprintanalysis—andtheseeffortswillpaycleardividendsforthecriminaljusticesystem.

Onedirectionistocontinuetoimprovelatentprintanalysisasasubjectivemethod.Withonlytwoblack-boxstudiessofar(withverydifferenterrorrates),thereisaneedforadditionalblack-boxstudiesbuildingonthestudydesignoftheFBIblack-boxstudy.Studiesshouldestimateerrorratesforlatentprintsofvaryingquality


81

andcompleteness,usingwell-definedmeasures.Asnotedabove,thestudiesshouldbedesignedandconductedinconjunctionwiththirdpartieswithnostakeintheoutcome.

Asecond—andmoreimportant—directionistoconvertlatentprintanalysisfromasubjectivemethodtoanobjectivemethod.Thepastdecadehasseenextraordinaryadvancesinautomatedimageanalysisbasedonmachinelearningandotherapproaches—leadingtodramaticimprovementsinsuchtasksasfacerecognition.295,296Inmedicine,forexample,itisexpectedthatautomatedimageanalysiswillbecomethegoldstandardformanyapplicationsinvolvinginterpretationofX-rays,MRIs,fundoscopy,anddermatologicalimages.297

Objectivemethodsbasedonautomatedimageanalysiscouldyieldmajorbenefits—includinggreaterefficiencyandlowererrorrates;itcouldalsoenableestimationoferrorratesfrommillionsofpairwisecomparisons.Initialeffortstodevelopautomatedsystemscouldnotoutperformhumans.298However,giventhepaceofprogressinimageanalysisandmachinelearning,webelievethatfullyautomatedlatentprintanalysisislikelytobepossibleinthenearfuture.Therehavealreadybeeninitialstepsinthisdirection,bothinacademiaandindustry.299

Themostimportantresourcetopropelthedevelopmentofobjectivemethodswouldbethecreationofhugedatabasescontainingknownprints,eachwithmanycorresponding”simulated”latentprintsofvaryingqualitiesandcompleteness,whichwouldbemadeavailabletoscientifically-trainedresearchersinacademiaandindustry.Thesimulatedlatentprintscouldbecreatedby“morphing”theknownprints,basedontransformationsderivedfromcollectionsofactuallatentprint-recordprintpairs.300

295See:cs.stanford.edu/people/karpathy/cvpr2015.pdf.296Lu,C.,andX.Tang.“Surpassinghuman-levelfaceverificationperformanceonLFWwithGaussianFace.”arxiv.org/abs/1404.3840(accessedJuly2,2016).Taigman,Y.,Yang,M.,Ranzato,M.,andL.Wolf.“Deepface:Closingthegaptohuman-levelperformanceinfaceverification.”www.cs.toronto.edu/~ranzato/publications/taigman_cvpr14.pdf(accessedJuly2,2016);andSchroff,F.,Kalenichenko,D.,andJ.Philbin.“FaceNet:Aunifiedembeddingforfacerecognitionandclustering.”arxiv.org/abs/1503.03832(accessedJuly2,2016).297Doi,K.“Computer-aideddiagnosisinmedicalimaging:historicalreview,currentstatusandfuturepotential.”ComputerizedMedicalImagingandGraphics,Vol.31,No.4-5(2007):198-211andShiraishi,J.,Li,Q.,Appelbaum,D.,andK.Doi.“Computer-aideddiagnosisandartificialintelligenceinclinicalimaging.”SeminarsinNuclearMedicine,Vol.41,No.6(2011):449-62.298Forexample,astudyin2010reportedthatthathumansoutperformedanautomatedprogramfortoolmarkcomparisons.SeeChumbley,L.S.etal.(2010).ValidationofToolMarkComparisonsObtainedUsingaQuantitative,Comparative,StatisticalAlgorithm.JournalofForensicSciences,55(4),pp.953-961.299Arunalatha,J.A.,Tejaswi,V.,Shaila,K.,Anvekar,D.,Venugopal,K.R.,Iyengar,S.S.,andL.M.Patnaik.“FIVDL:FingerprintImageVerificationusingDictionaryLearning.”ProcediaComputerScience,Vol.54(2015):482-490andSrihari,S.N.“QuantitativeMeasuresinSupportofLatentPrintComparison:FinalTechnicalReport.”NIJAwardNumber:2009-DN-BX-K208,UniversityatBuffalo,SUNY,2013.www.crime-scene-investigator.net/QuantitativeMeasuresinSupportofLatentPrint.pdf.Inaddition,ChristopheChampod’sgroupatUniversitédeLausannehasanactiveprograminthisarea.300Forprivacy,fingerprintsfromdeceasedindividualscouldbeused.


82

5.5FirearmsAnalysis

Methodology

Infirearmsanalysis,examinersattempttodeterminewhetherammunitionisorisnotassociatedwithaspecificfirearmbasedontoolmarksproducedbygunsontheammunition.301,302(Briefly,gunbarrelsaretypicallyrifledtoimproveaccuracy,meaningthatspiralgroovesarecutintothebarrel’sinteriortoimpartspinonthebullet.Randomindividualimperfectionsproducedduringthetool-cuttingprocessandthrough“wearandtear”ofthefirearmleavetoolmarksonbulletsorcasingsastheyexitthefirearm.)

Thedisciplineisbasedontheideathatthetoolmarksproducedbydifferentfirearmsvarysubstantiallyenough(owingtovariationsinmanufactureanduse)toallowcomponentsoffiredcartridgestobeidentifiedwithparticularfirearms.Forexample,examinersmaycompare“questioned”cartridgesfromanunknownguntotestfiresfromagunbelongingtoaknownindividual.

Briefly,examinationbeginswithanevaluationofclasscharacteristicsofthebulletsandcasings,whicharefeaturesthatarepermanentandpredeterminedbeforemanufacture.Iftheseclasscharacteristicsaredifferent,aneliminationconclusionisrendered.Iftheclasscharacteristicsaresimilar,theexaminationproceedstoidentifyandcompareindividualcharacteristics,suchasthestriathatariseduringfiringfromaparticulargun.AccordingtotheAssociationofFirearmandToolMarkExaminers(AFTE)the“mostwidelyacceptedmethodusedinconductingatoolmarkexaminationisaside-by-side,microscopiccomparisonofthemarkingsonaquestionedmaterialitemtoknownsourcemarksimpartedbyatool.”303

Background

Inthepreviouschapter(p.41),PCASTexpressedconcernsaboutcertainfoundationaldocumentsunderlyingthescientificdisciplineoffirearmandtoolmarkexamination.Inparticular,weobservedthatAFTE’s“TheoryofIdentificationasitRelatestoToolmarks”—whichdefinesthecriteriaformakinganidentification—iscircular.304The“theory”statesthatanexaminermayconcludethattwoitemshaveacommonoriginiftheirmarksarein“sufficientagreement,”where“sufficientagreement”isdefinedastheexaminerbeingconvincedthattheitemsareextremelyunlikelytohaveadifferentorigin.Inaddition,the“theory”explicitlystatesthatconclusionsaresubjective.

301Examinerscanalsoundertakeotherkindsofanalysis,suchasfordistancedeterminations,operabilityoffirearms,andserialnumberrestorationsaswellastheanalyzeprimerresiduetodeterminewhethersomeonerecentlyhandledaweapon.302Formorecompletedescriptions,see,e.g.,NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009),andhttps://archives.fbi.gov/archives/about-us/lab/forensic-science-communications/fsc/july2009/review/2009_07_review01.htm.303See:FoundationalOverviewofFirearm/ToolmarkIdentificationtabonafte.org/resources/swggun-ark(accessedMay12,2016).304AssociationofFirearmandToolMarkExaminers.“TheoryofIdentificationasitRelatestoToolMarks:Revised,”AFTEJournal,Vol.43,No.4(2011):287.


83

Muchattentioninthisscientificdisciplinehasfocusedontryingtoprovethenotionthateverygunproduces“unique”toolmarks.In2004,theNIJaskedtheNRCtostudythefeasibility,accuracy,reliability,andadvisabilityofdevelopingacomprehensivenationalballisticsdatabaseofimagesfrombulletsfiredfromall,ornearlyall,newlymanufacturedorimportedgunsforthepurposeofmatchingballisticsfromacrimescenetoagunandinformationonitsinitialowner.

Inits2008report,anNRCcommittee,respondingtoNIJ’srequest,foundthat“thevalidityofthefundamentalassumptionsofuniquenessandreproducibilityoffirearms-relatedtoolmarks”hadnotyetbeendemonstratedandthat,givencurrentcomparisonmethods,adatabasesearchwouldlikely“returntoolargeasubsetofcandidatematchestobepracticallyusefulforinvestigativepurposes.”305

Ofcourse,itisnotnecessarythattoolmarksbeuniqueforthemtoprovideusefulinformationwhetherabulletmayhavebeenfiredfromaparticulargun.However,itisessentialthattheaccuracyofthemethodforcomparingthembeknownbasedonempiricalstudies.

Firearmsanalystshavelongstatedthattheirdisciplinehasnear-perfectaccuracy.Ina2009article,thechiefoftheFirearms-ToolmarksUnitoftheFBILaboratorystatedthat“aqualifiedexaminerwillrarelyifevercommitafalse-positiveerror(misidentification),”citinghisreview,inanaffidavit,ofempiricalstudiesthatshowedvirtuallynoerrors.306

Withrespecttofirearmsanalysis,the2009NRCreportconcludedthat“sufficientstudieshavenotbeendonetounderstandthereliabilityandreproducibilityofthemethods”—thatis,thatthefoundationalvalidityofthefieldhadnotbeenestablished.307

TheScientificWorkingGrouponFirearmsAnalysis(SWGGUN)respondedtothecriticismsinthe2009NRCreportbystatingthat:

TheSWGGUNhasbeenawareofthescientificandsystemicissuesidentifiedinthisreportforsometimeandhasbeenworkingdiligentlytoaddressthem....[theNRCreport]identifiestheareaswherewemustfundamentallyimproveourprocedurestoenhancethequalityandreliabilityofourscientificresults,aswellasbetterarticulatethebasisofourscience.308

305NationalResearchCouncil.BallisticImaging.TheNationalAcademiesPress.WashingtonDC.(2008)p.3-4.306See:www.fbi.gov/about-us/lab/forensic-science-communications/fsc/july2009/review/2009_07_review01.htm.307Thereportstatesthat“Toolmarkandfirearmsanalysissuffersfromthesamelimitationsdiscussedaboveforimpressionevidence.Becausenotenoughisknownaboutthevariabilitiesamongindividualtoolsandguns,wearenotabletospecifyhowmanypointsofsimilarityarenecessaryforagivenlevelofconfidenceintheresult.Sufficientstudieshavenotbeendonetounderstandthereliabilityandrepeatabilityofthemethods.Thecommitteeagreesthatclasscharacteristicsarehelpfulinnarrowingthepooloftoolsthatmayhaveleftadistinctivemark.”NationalResearchCouncil.AcademyofSciences,(2009)StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(p.154)308See:www.swggun.org/index.php?option=com_content&view=article&id=37&Itemid=22.


84

Earlystudiesofthevalidityoffirearmsanalysis

Becausefirearmsanalysisisatpresentasubjectivefeature-comparisonmethod,itsfoundationalvaliditycanonlybeestablishedthroughmultipleindependentblackboxstudies,asdiscussedabove.

Althoughfirearmsanalysishasbeenusedformanydecades,onlyrelativelyrecentlyhasitsvaliditybeensubjectedtomeaningfulempiricaltesting.Overthepast15years,thefieldhasundertakenanumberofstudiesthathavesoughttoestimatetheaccuracyofexaminers’conclusions.Whiletheresultsdemonstratethatexaminerscanundersomecircumstancesidentifythesourceoffiredammunition,manyofthestudieswerenotappropriateforassessingscientificvalidityandestimatingthereliabilitybecausetheyemployedartificialdesignsthatdifferinimportantwaysfromtheproblemsfacedincasework.

Specifically,manyofthestudiesaskedexaminerstoperformallpairwisecomparisonswithinorbetweensmallsamplessets.Becausetheset-basedcomparisonsarenotindependentofoneanother,examinershavetheopportunitytousepartialinformationtomakeinferencesaboutthestudydesignandaboutspecificanswers.(TheDirectoroftheDefenseForensicScienceCenteranalogizedthesituationtosolvinga“Sudoku”puzzle,whereinitialanswerscanbeusedtofillinadditionalanswers.309)Moreover,theempiricalresultsoftheseset-basedstudiessuggestthattheymaysubstantiallyunderestimatethefalsepositiverate.

Wediscussthemostwidelycitedofthesestudiesbelow.Weadoptthesameframeworkasforlatentprints,focusingprimarilyon(1)the95%upperconfidencelimitofthefalsepositiverateand(2)falsepositiveratesbasedontheproportionofconclusiveexaminations,astheappropriatemeasurestoreport(seep.70).

Within-setcomparisonSomestudieshavepresentedexaminerswith,forexample,acollectionofsamplesandaskedthemtodeterminewhichsampleswerefiredfromthesamefirearm.Wereviewedtwosuchstudies.310,311Inthesestudies,mostofthesampleswerefromdistinctsources,witheither2or3fromthesamesource.Acrossthetwostudies,examinersidentified55of61matchesandmadenofalsepositives.Inthefirststudy,thevastmajorityofdifferent-sourcesamples(97%)weredeclaredinconclusive;therewereonly18conclusiveexaminationsfor

309PCASTinterviewwithJeffSalyards,Director,DFSC.310Smith,E.“Cartridgecaseandbulletcomparisonvalidationstudywithfirearmssubmittedincasework.”AFTEJournal,Vol.37,No.2(2005):130-5.InthisstudyfromtheFBI,cartridgesandbulletswerefiredfromnineRugerP89pistolsfromcasework.Examinersweregivenpackets(ofcartridgesorbullets)containingsamplesfiredfromeachofthe9gunsandoneadditionalsamplefiredfromoneoftheguns;theywereaskedtodeterminewhichsampleswerefiredfromthesamegun.Amongthe16same-sourcecomparisons,therewere13identificationsand3inconclusives.Amongthe704different-sourcecomparisons,97%weredeclaredinconclusives,2.5%weredeclaredexclusionsand0%falsepositives.311DeFrance,C.S.,andM.D.VanArsdale.“Validationstudyofelectrochemicalrifling.”AFTEJournal,Vol.35,No.1(2003):35-7.InthisstudyfromtheFBI,bulletswerefiredfrom5consecutivelymanufacturedSmith&Wesson.357Magnumcaliberriflebarrels.Eachof9examinersreceivedtwotestpackets,eachcontainingabulletfromeachofthe5gunsandtwoadditionalbullets(fromthedifferentgunsinonepacket,fromthesamegunintheother);theywereaskedtoperformall42possiblepairwisecomparisons,whichincluded37different-sourcecomparisons.Ofthe45totalsame-sourcecomparisons,therewere42identificationsand3inconclusives.Forthe333totaldifferent-sourcecomparisons,thepaperstatesthattherewerenofalsepositives,butdoesnotreportthenumberofinconclusiveexaminations.


85

different-sourcecartridgesandnoconclusiveexaminationsfordifferent-sourcebullets.312Inthesecondstudy,theresultsareonlydescribedinbriefparagraphandthenumberofconclusiveexaminationsfordifferent-sourcepairswasnotreported.Itisthusimpossibletoestimatethefalsepositiverateamongconclusiveexaminations,whichisthekeymeasureforconsideration(asdiscussedabove).

Set-to-setcomparison/closedsetAcommondesignhasbeen“set-to-setcomparisonswithaclosedset.”Inthiscase,examinersaregivenasetofquestionedsamplesandaskedtocomparethemtoasetofknownstandards,representingthepossiblegunsfromwhichthequestionedammunitionhadbeenfired.Inthe“closedset”design,thesourcegunwasalwayspresent.Weanalyzedfoursuchstudiesindetail.313,314,315,316Inthesestudies,examinersweregivenacollectionofquestionedbulletsand/orcartridgecasesfiredfromasmallnumberofconsecutivelymanufacturedfirearms(3,10,10,and10guns,respectively)andacollectionofbullets(orcasings)knowntohavebeenfiredfromthesesameguns.Theywerethenaskedtoperformamatchingexercise—assigningthebullets(orcasings)inonesettothebullets(orcasings)intheotherset.

This“closedset”designissimplerthantheproblemencounteredincasework,becausethecorrectanswerisalwayspresentinthecollection.Insuchstudies,examinerscanperformperfectlyiftheysimplymatcheachbullettothestandardthatisclosest.Bycontrast,inanopensetstudy(asincasework),thereisnoguaranteethatthecorrectsourceispresent—andthusnoguaranteethattheclosestmatchiscorrect.Closedsetcomparisonswouldthusbeexpectedtounderestimatethefalsepositiverate.

312Somelaboratorypoliciesmandateaveryhighbarfordeclaringexclusions.313Stroman,A.“Empiricallydeterminedfrequencyoferrorincartridgecaseexaminationsusingadeclareddouble-blindformat.”AFTEJournal,Vol.46,No.2(2014):157-175.Inthisstudy,bulletswerefiredfrom3Smith&Wessonguns.Eachof25examinersreceivedatestsetcontaining3questionedcartridgesand3knowncartridgesfromeachgun.Ofthe75answersreturned,therewere74correctassignmentsandoneinconclusiveexamination.314Brundage,D.J.“Theidentificationofconsecutivelyrifledgunbarrels.”AFTEJournal,Vol.30,No.3(1998):438-44.Inthisstudy,bulletswerefiredfrom10consecutivelymanufactured9millimeterRugerP-85semi-automaticpistolbarrels.Eachof30examinersreceivedatestsetcontaining20questionedbulletstocomparetoasetof15standards,containingatleastonebulletfiredfromeachofthe10guns.Ofthe300answersreturned,therewerenoincorrectassignmentsandoneinconclusiveexamination.315Fadul,T.G.,Hernandez,G.A.,Stoiloff,S.,andS.Gulati.“Anempiricalstudytoimprovethescientificfoundationofforensicfirearmandtoolmarkidentificationutilizing10consecutivelymanufacturedslides.”AFTEJournal.Vol.45,No.4(2013):376-93.Anempiricalstudytoimprovethescientificfoundationofforensicfirearmandtoolmarkidentificationutilizing10consecutivelymanufacturedslides.Inthisstudy,bulletswerefiredfrom10consecutivelymanufacturedsemi-automatic9mmRugerpistolslides.Eachof217examinersreceivedatestsetconsistingof15questionedcasingsand2knowncartridgesfromeachofthe10guns.Ofthe3255answersreturned,therewere3239correctassignments,14inconclusiveexaminationsand2falsepositives.316Hamby,J.E.,Brundage,D.J.,andJ.W.Thorpe.“Theidentificationofbulletsfiredfrom10consecutivelyrifled9mmRugerpistolbarrels:aresearchprojectinvolving507participantsfrom20countries.”AFTEJournal,Vol.41,No.2(2009):99-110.Inthisstudy,bulletswerefiredfrom10consecutivelyrifledRugerP-85barrels.Eachof440examinersreceivedatestsetconsistingof15questionedcasingsand2knowncartridgesfromeachofthe10guns.Ofthe6600answersreturned,therewere6593correctassignments,7inconclusiveexaminationsandnofalsepositives.


86

Importantly,itisnotnecessarythatexaminersbetoldexplicitlythatthestudydesigninvolvesaclosedset.Asoneofthestudiesnoted,

Theparticipantswerenottoldwhetherthequestionedcasingsconstitutedanopenorclosedset.However,fromthequestionnaire/answersheet,participantscouldhaveassumeditwasaclosedsetandthateveryquestionedcasingshouldbeassociatedwithoneofthetenslides.317

Moreover,theparticipants’surmisewilltendtobeconfirmedastheyfindthatmanyofthequestionedcasingshavesimilaritiestotheknowncasings.Clearevidencethatparticipantscaninferthatthestudiesinvolveaclosed-setdesignisthatthefalsepositiverateintheclosed-setstudieswasroughly100-foldlowerthaninsomerecentstudiesdescribedbelowthatinvolvedapartlyopenorfullyopendesign(Table2).318

Inshort,theclosed-setdesignisproblematicinprincipleandappearstounderestimatethefalsepositiverateinpractice.319Thedesignisnotappropriateforassessingscientificvalidityandmeasuringreliability.

Set-to-setcomparison/partlyopenset(‘MiamiDadestudy’)Onestudyinvolvedaset-to-setcomparisoninwhichafewofthequestionedsampleslackedamatchingknownstandard.320The165examinersinthestudywereaskedtoassignacollectionof15questionedsamples,firedfrom10pistols,toacollectionofknownstandards;twoofthe15questionedsamplescamefromagunforwhichknownstandardswerenotprovided.Forthesetwosamples,therewere188eliminations,138inconclusivesand4falsepositives.Theinconclusiveratewas41.8%andthefalsepositiverateamongconclusiveexaminationswas2.1%(confidenceinterval0.6%-5.25%).Thefalsepositiveratecorrespondstoanestimatedrateof1errorin48cases,withupperboundbeing1in19.

Comparedtotheclosed-setstudiesabove,theresultsforthesetwosamplesarenotable:(1)theproportionofinconclusiveresultswas200-foldhigherand(2)thefalsepositiveratewasroughly100-foldhigher.

Recentblack-boxstudyoffirearmsanalysis

In2011,theForensicResearchCommitteeofASCLDidentified,amongthehighestrankedneedsinforensicscience,theimportanceofundertakingablack-boxstudyinfirearmsanalysisanalogoustotheFBI’sblack-box

317Fadul,T.G.,Hernandez,G.A.,Stoiloff,S.,andS.Gulati.“Anempiricalstudytoimprovethescientificfoundationofforensicfirearmandtoolmarkidentificationutilizing10consecutivelymanufacturedslides.”AFTEJournal,Vol.45,No.4(2013):376-93.318Ofthe10,230answersreturnedacrossthethreestudies,thereweretherewere10,205correctassignments,23inconclusiveexaminationsand2falsepositives.319Stroman(2014)acknowledgesthat,althoughthetestinstructionsdidnotexplicitlyindicatewhetherthestudywasclosed,theirstudycouldbeimprovedif“additionalfirearmswereusedandknownsfromonlyaportionofthosefirearmswereusedinthetestkits,thuspresentinganopensetofunknownstotheparticipants.Whilethiscouldincreasethechancesofinconclusiveresults,itwouldbeamoreaccuratereflectionofthetypesofevidencereceivedinrealcasework.”320Fadul,T.G.,Hernandez,G.A.,Stoiloff,S.,andS.Gulati.“AnempiricalstudytoimprovethescientificfoundationofforensicfirearmandtoolmarkidentificationutilizingconsecutivelymanufacturedGlockEBISbarrelswiththesameEBISpattern.”NationalInstituteofJusticeGrant#2010-DN-BX-K269,December2013.www.ncjrs.gov/pdffiles1/nij/grants/244232.pdf.


87

studyoflatentfingerprints.DFSC,dissatisfiedwiththedesignofpreviousstudiesoffirearmsanalysis,concludedthatablack-boxstudywasneededandshouldbeconductedbyanindependenttestinglaboratoryunaffiliatedwithlawenforcementthatwouldengageforensicexaminersasparticipantsinthestudy.DFSCandDefenseForensicsandBiometricsAgencydecidedtofundastudybytheAmesLaboratory,aDepartmentofEnergynationallaboratoryaffiliatedwithIowaStateUniversity.321,322

Independenttests/open(‘AmesLaboratorystudy’)ThestudyemployedasimilardesigntotheFBI’sblack-boxstudyoflatentfingerprints,withmanyexaminersmakingaseriesofindependentcomparisondecisionsbetweenaquestionedsampleandoneormoreknownsamplesthatmayormaynotcontainthesource.Thesamplesallcamefrom25newlypurchased9mmRugerpistols.323Eachof218examiners324waspresentedwith15separatecomparisonproblems—eachconsistingofonequestionedsampleandthreeknowntestfiresfromthesameknowngun,whichmightormightnothavebeenthesource.325Unbeknownsttotheexaminers,therewerefivesame-sourceandtendifferent-sourcecomparisons.(Inanidealdesign,theproportionofsame-anddifferent-sourcecomparisonswoulddifferamongexaminers.)

Amongthe2178different-sourcecomparisons,therewere1421eliminations,735inconclusivesand22falsepositives.Theinconclusiveratewas33.7%andthefalsepositiverateamongconclusiveexaminationswas1.5%(upper95%confidenceinterval2.2%).Thefalsepositiveratecorrespondstoanestimatedrateof1errorin66cases,withupperboundbeing1in46.(Itshouldbenotedthat20ofthe22falsepositivesweremadebyjust5ofthe218examiners—stronglysuggestingthatthefalsepositiverateishighlyheterogeneousacrosstheexaminers.)

TheresultsforthevariousstudiesareshowninTable2.Thetablesshowastrikingdifferencebetweentheclosed-setstudies(whereamatchingstandardisalwayspresentbydesign)andthenon-closedstudies(wherethereisnoguaranteethatanyoftheknownstandardsmatch).Specifically,theclosed-setshowadramaticallylowerrateofinconclusiveexaminationsandoffalsepositives.Withthisunusualdesign,examinerssucceedinansweringallquestionsandachieveessentiallyperfectscores.Inthemorerealisticopendesigns,theseratesaremuchhigher.

Table2:Resultsfromfirearmsstudies*

321Baldwin,D.P.,Bajic,S.J.,Morris,M.,andD.Zamzow.“Astudyoffalse-positiveandfalse-negativeerrorratesincartridgecasecomparisons.”AmesLaboratory,USDOE,TechnicalReport#IS-5207(2014)afte.org/uploads/documents/swggun-false-postive-false-negative-usdoe.pdf.322[Addtext]Smith,TashaP.,G.AndrewSmith,andJeffreyB.Snipes."Avalidationstudyofbulletandcartridgecasecomparisonsusingsamplesrepresentativeofactualcasework."Journalofforensicsciences(2016).323Onecriticism,raisedbyaforensicscientist,isthatthestudydidnotinvolveconsecutivelymanufacturedguns.324ParticipantsweremembersofAFTEwhowerepracticingexaminersemployedbyorretiredfromanationalorinternationallawenforcementagency,withsuitabletraining.325Actualcaseworkmayinvolvemorecomplexsituations(forexample,manydifferentbulletsfromacrimescene).But,aproperassessmentoffoundationalvaliditymuststartwiththequestionofhowoftenanexaminercandeterminewhetheraquestionedbulletcomesfromaspecificknownsource.


88

StudyType Resultsfordifferent-sourcecomparisons

RawData Inconclusives Falsepositivesamongconclusiveexams326

Exclusions/Inconclusives/Falsepositives

Freq.(ConfidenceBound)

EstimatedRate

BoundonRate

Set-to-set/closed 10,205/23/2 0.2% 0.02%(0.06%) 1in5103 1in1612

Set-to-set/slightlyopen 188/138/4 41.8% 2.0%(4.7%) 1in49 1in21

Independent/open(Black-boxstudy)

1421/735/22 33.7% 1.5%(2.2%) 1in66 1in46

*“Inconclusives”:Proportionoftotalexaminationsthatwerecalledinconclusive.“RawData”:Numberoffalsepositivesdividedbynumberofconclusiveexaminationsinvolvingquestioneditemswithoutacorrespondingknown(forset-to-set/slightlyopen)ornon-matedpairs(forindependent/open).“Freq.(ConfidenceBond)”:Pointestimateoffalsepositivefrequency,withtheupper95%confidencebounds.“Estimated”:Theoddsofafalsepositiveoccurring,basedontheobservedproportionoffalsepositives.“Bound”:Theoddsofafalsepositiveoccurring,basedontheupperboundoftheconfidenceinterval—thatis,theratecouldreasonablybeashighasthisvalue.

Conclusions

Theearlystudiesindicatethatexaminerscan,undersomecircumstances,associateammunitionwiththegunfromwhichitwasfired.However,asdescribedabove,mostofthesestudiesinvolveddesignsthatarenotappropriateforassessingthescientificvalidityorestimatingthereliabilityofthemethodaspracticed.Indeed,comparisonofthestudiessuggeststhat,becauseoftheirdesign,manyfrequentlycitedstudiesseriouslyunderestimatethefalsepositiverate.

Atpresent,thereisonlyasinglestudythatwasappropriatelydesignedtotestfoundationalvalidityandestimatereliability(AmesLaboratorystudy).Importantly,thestudywasconductedbyanindependentgroup,unaffiliatedwithacrimelaboratory.Althoughthereportisavailableontheweb,ithasnotyetbeensubjectedtopeerreviewandpublication.

Thescientificcriteriaforfoundationalvalidityrequireappropriatelydesignedstudiesbymorethanonegrouptoensurereproducibility.Becausetherehasbeenonlyasingleappropriatelydesignedstudy,thecurrentevidencefallsshortofthescientificcriteriaforfoundationalvalidity.Thereisthusaneedforadditional,appropriatelydesignedblack-boxstudiestoprovideestimatesofreliability.

Finding6:Firearmsanalysis

Foundationalvalidity.PCASTfindsthatfirearmsanalysiscurrentlyfallsshortofthecriteriaforfoundationalvalidity,becausethereisonlyasingleappropriatelydesignedstudytomeasurevalidityandestimatereliability.Thescientificcriteriaforfoundationalvalidityrequiremorethanonesuchstudy,todemonstratereproducibility.

326Theratesforallexaminationsare,readingacrossrows:1in5115,1in1416,1in83,1in33,1in99,and1in66.


89

Whetherfirearmsanalysisshouldbedeemedadmissiblebasedoncurrentevidenceisadecisionthatbelongstothecourts.

Iffirearmsanalysisisallowedincourt,thescientificcriteriaforvalidityasappliedshouldbeunderstoodtorequireclearlyreportingtheerrorratesseeninappropriatelydesignedblack-boxstudies(estimatedat1in66,witha95%confidencelimitof1in46,intheonesuchstudytodate).

Validityasapplied.Iffirearmsanalysisisallowedincourt,validityasappliedwould,fromascientificstandpoint,requirethattheexpert:

(1)hasundergonerigorousproficiencytestingonalargenumberoftestproblemstomeasurehisorheraccuracyanddisclosestheresultsoftheproficiencytesting;and

(2)discloseswhether,whenperformingtheexamination,heorshewasawareofanyotherfactsofthecasethatmightinfluencetheconclusion.

ThePathForward

Continuingeffortsareneededtoimprovethestateoffirearmsanalysis—andtheseeffortswillpaycleardividendsforthecriminaljusticesystem.

Onedirectionistocontinuetoimprovefirearmsanalysisasasubjectivemethod.Withonlyoneblack-boxstudysofar,thereisaneedforadditionalblack-boxstudiesbasedonthestudydesignoftheAmesLaboratoryblack-boxstudy.Asnotedabove,thestudiesshouldbedesignedandconductedinconjunctionwiththirdpartieswithnostakeintheoutcome(suchastheAmesLaboratoryorresearchcenterssuchastheCenterforStatisticsandApplicationsinForensicEvidence(CSAFE)).Thereisalsoaneedformorerigorousproficiencytestingofexaminers,usingproblemsthatareappropriatelychallengingandpublicallydisclosedafterthetest.Asecond—andmoreimportant—directionis(aswithlatentprintanalysis)toconvertfirearmsanalysisfromasubjectivemethodtoanobjectivemethod.

Thiswouldinvolvedevelopingandtestingimage-analysisalgorithmsforcomparingthesimilarityoftoolmarksonbullets.Therehavealreadybeenencouragingstepstowardthisgoal.327Recenteffortstocharacterize3Dimagesofbulletshaveusedstatisticalandmachinelearningmethodstoconstructaquantitative“signature”foreachbulletthatcanbeusedforcomparisonsacrosssamples.Arecentreviewdiscussesthepotentialforsurface

327Forexample,arecentstudyuseddatafromthree-dimensionalconfocalmicroscopyofammunitiontodevelopasimilaritymetrictocompareimages.Byperformingallpairwisecomparisonsamongatotalof90cartridgesfiredfrom10pistolslides,theauthorsfoundthatthedistributionofthemetricforsame-gunpairsdidnotoverlapthedistributionofthemetricfordifferent-gunpairs.Althoughasmallstudy,itisencouraging.Weller,T.J.,Zheng,X.A.,Thompson,R.M.,andF.Tulleners.“Confocalmicroscopyanalysisofbreechfacemarksonfiredcartridgecasesfrom10consecutivelymanufacturedpistolslides.”JournalofForensicSciences,Vol.57,No.4(2012):912-17.


90

topographicmethodsinballisticsandsuggestsapproachestousethesemethodsinfirearmsexamination.328Theauthorsnotethatthedevelopmentofopticalmethodshaveimprovedthespeedandaccuracyofcapturingsurfacetopography,leadingtoimprovequantificationofthedegreeofsimilarity.Inarecentstudy,researchersusedimagesfromanearlierstudytodevelopacomputer-assistedapproachtomatchbulletsthatminimizeshumaninput.329Thegroup’salgorithmextractsaquantitativesignaturefromabullet3Dimage,comparesthesignatureacrosstwoormoresamples,andproducesa“matchingscore”,thatreflectsthestrengthofthematch.Onthesmalltestdataset,thealgorithmhadaverylowerrorrate.Thereareadditionaleffortsintheprivatesectorfocusedondevelopmentofaccuratehigh-resolutioncartridgecasingrepresentationstoimproveaccuracyandallowforhigherqualityscoringfunctionstoimproveandassignmatchconfidenceduringdatabasesearches.ThecurrentNIBINdatabaseusesolder(non-3D)technologyanddoesnotprovideascoringfunctionorconfidenceassignmenttoeachcandidatematch.Ithasbeensuggestedthatascoringfunctioncouldbeusedforblindverificationforhumanexaminers.Giventhetremendousprogressoverthepastdecadeinotherfieldsofimageanalysis,webelievethatfullyautomatedfirearmsanalysisislikelytobepossibleinthenearfuture.However,effortsarecurrentlyhamperedbylackofaccesstorealisticallylargeandcomplexdatabasesthatcanbeusedtocontinuedevelopmentofthesemethodsandvalidateinitialproposals.NIST,incoordinationwiththeFBILaboratory,shouldplayaleadershiproleinpropellingthistransformationbycreatinganddisseminatingappropriatelargedatasets.Theseagenciesshouldalsoprovidegrantsandcontractstosupportwork—andsystematicprocessestoevaluatemethods.Inparticular,webelievethat“prize”competitions—basedonlarge,publiclyavailablecollectionsofimages330—couldattractsignificantinterestfromacademicandindustry.

5.6FootwearAnalysis:IdentifyingCharacteristicsMethodology

Footwearanalysisisaprocessthattypicallyinvolvescomparingaknownobject,suchasashoe,toacompleteorpartialimpressionfoundatacrimescene,toassesswhethertheobjectislikelytobethesourceoftheimpression.Theprocessproceedsinastepwisemanner,beginningwithacomparisonof“classcharacteristics”(suchasdesign,physicalsize,andgeneralwear)andthenmovingto“identifyingcharacteristics”or“randomly

328Vorburger,T.V.,Song,J.,andN.Petraco.“Toporgraphymeasurementsandapplicationsinballisticsandtoolmarkidentification.”Surfacetopography:MetrologyandProperties,Vol.4(2016)013002.329Hareetal.http://arxiv.org/abs/1601.05788.330OnJuly7,2016NISTreleasedtheNISTBallisticsToolmarkResearchDatabase(NBTRD)asanopen-accessresearchdatabaseofbulletandcartridgecasetoolmarkdata(tsapps.nist.gov/NRBTD).Thedatabasecontainsreflectancemicroscopyimagesandthree-dimensionalsurfacetopographydataacquiredbyNISTorsubmittedbyusers.


91

acquiredcharacteristics(RACs)”(suchasmarksonashoecausedbycuts,nicks,andgougesinthecourseofuse).331

Inthisreport,wedonotaddressthequestionofwhetherexaminerscanreliablydetermineclasscharacteristics—forexample,whetheraparticularshoeprintwasmadebyasize12shoeofaparticularmake.Whileitisimportantthatthatstudiesbeundertakentoestimatethereliabilityoffootwearanalysisaimedatdeterminingclasscharacteristics,PCASTchosenottofocusonthisaspectoffootwearexaminationbecauseitisnotinherentlyachallengingmeasurementproblemtodetermineclasscharacteristics,toestimatethefrequencyofshoeshavingaparticularclasscharacteristic,or(forjurors)tounderstandthenatureofthefeaturesinquestion.

Instead,PCASTfocusedonthereliabilityofconclusions,basedonRACs,thatanimpressionwaslikelytohavecomefromaspecificpieceoffootwear.Thisisamuchharderproblem,becauseitrequiresknowinghowaccuratelyexaminersidentifyspecificfeaturessharedbetweenashoeandanimpression,howoftentheyfailtoidentifyfeaturesthatwoulddistinguishthem,andwhatprobativevalueshouldbeascribedtoaparticularRAC.

Despitetheabsenceofempiricalstudiesthatmeasureexaminers’accuracy,authoritiesinthefootwearfieldexpressconfidencethattheycanidentifythesourceofanimpressionbasedonasingleRAC.

Asdescribedina2009articlebyanFBIforensicexaminerpublishedintheFBI’sForensicScienceCommunications:

Anexaminerfirstdetermineswhetheracorrespondenceofclasscharacteristicsexistsbetweenthequestionedfootwearimpressionandtheknownshoe.Iftheexaminerdeemsthattherearenoinconsistenciesinclasscharacteristics,thentheexaminationprogressestoanyidentifyingcharacteristicsinthequestionedimpression.Theexaminercomparesthesecharacteristicswithanyidentifyingcharacteristicsobservedontheknownshoe.Althoughunpredictableintheiroccurrence,thesize,shape,andpositionofthesecharacteristicshavealowprobabilityofrecurrenceinthesamemanneronadifferentshoe.Thus,combinedwithclasscharacteristics,evenoneidentifyingcharacteristicisextremelypowerfulevidencetosupportaconclusionofidentification.332

Insupport,thearticlecitesaleadingtextbookonfootwearidentification:

AccordingtoWilliamJ.Bodziak(2000),“Positiveidentificationsmaybemadewithasfewasonerandomidentifyingcharacteristic,butonlyifthatcharacteristicisconfirmable;hassufficientdefinition,clarity,and

331See:SWGTREADRangeofConclusionsStandardsforFootwearandTireImpressionExaminations(2013).SWGTREADGuidefortheExaminationofFootwearandTireImpressionEvidence(2006).BodziakW.J.FootwearImpressionEvidence:Detection,Recovery,andExamination.2nded.CRCPress-Taylor&Francis,BocaRaton,Florida(2000):p347.332Smith,M.B.TheForensicAnalysisofFootwearImpressionEvidence.www.fbi.gov/about-us/lab/forensic-science-communications/fsc/july2009/review/2009_07_review02.htm


92

features;isinthesamelocationandorientationontheshoeoutsole;andintheopinionofanexperiencedexaminer,wouldnotoccuragainonanothershoe.”333

ThearticlepointstoamathematicalmodelbyStonethatclaimsthatthechanceis1in16,000thattwoshoeswouldshareoneidentifyingcharacteristicsand1in683billionthattheywouldsharethreecharacteristics.334

Suchclaimsfor“identification”basedonfootwearanalysisarebreathtaking—butlackscientificfoundation.

ThestatementbyBodziakhastwocomponents:(i)thattheexaminerconsistentlyobservesademonstrableRACinasetofimpressions;and(ii)thattheexaminerispositivethattheRACwouldnotoccuronanothershoe.Thefirstisnotunreasonable,butthepartisdeeplyproblematic:Itrequirestheexaminertorelyonrecollectionsandguessesaboutthefrequencyoffeatures. ThemodelbyStoneisentirelytheoretical:itmakesmanyunsupportedassumptions(aboutthefrequencyandstatisticalindependenceofmarks)thatitdoesnottestinanyway.

Theentireprocess—fromchoiceoffeaturestoinclude(andignore)andthedeterminationofrarity—reliesentirelyonanexaminer’ssubjectivejudgment.Undersuchcircumstances,itisessentialthatthescientificvalidityofthemethodandestimatesofitsreliabilitybeestablishedbymultiple,appropriateblack-boxstudies.335

Background

The2009NRCreportcitedsomepapersthatcastdoubtonwhetherfootwearexaminersreachconsistentconclusionswhenpresentedwiththesameevidence.Forexample,thereportcontainedadetaileddiscussionofa1996Europeanpaperthatpresentedexaminerswithsixmockcases—twoinvolvingwornshoesfromcrimescenes,fourwithnewshoesinwhichspecificidentifyingcharacteristicshadbeendeliberatelyadded;thepaperreportedconsiderablevariationintheiranswers.336PCASTalsonotesa1999Israelistudyinvolvingtwocasesfromcrimescenesthatreachedsimilarconclusions.337

Inresponsetothe2009NRCreport,a2013paperclaimedtodemonstratethatAmericanandCanadianfootwearanalystsexhibitgreaterconsistencythanseeninthe1996Europeanstudy.338However,thisstudydifferedsubstantiallybecausetheexaminersinthisstudydidnotconducttheirownexaminations.Forexample,

333BodziakW.J.FootwearImpressionEvidence:Detection,Recovery,andExamination.2nded.CRCPress-Taylor&Francis,BocaRaton,Florida(2000).334Stone,R.S.“Footwearexaminations:Mathematicalprobabilitiesoftheoreticalindividualcharacteristics.”JournalofForensicIdentification,Vol.56,No.4(2006):577-99.335Inadditiontoblack-boxstudies,white-boxstudiesarealsovaluabletoidentifythesourcesoferrors.336Majamma,H.,andA.Ytti.“Surveyoftheconclusionsdrawnofsimilarfootwearcasesinvariouscrimelaboratories.”ForensicScienceInternational.Vol.82,No.1(1996):109-20.337Shor,Y.,andS.Weisner.“Surveyontheconclusionsdrawnonthesamefootwearmarksobtainedinactualcasesbyseveralexpertsthroughouttheworld.”JournalofForensicScience,Vol.44,No.2(1999):380-4384.338Hammer,L.,Duffy,K.,Fraser,J.,andN.N.Daeid.“Astudyofthevariabilityinfootwearimpressioncomparisonconclusions.”JournalofForensicIdentification,Vol.63,No.2(2013):205-18.


93

thephotographswerepre-annotatedtocalloutallrelevantfeaturesforcomparison—thatis,theexaminerswerenotaskedtoidentifythefeatures.339Thus,thestudy,byvirtueofitsdesign,cannotaddresstheconsistencyoftheexaminationprocess.

Moreover,thefundamentalissueisnotoneofconsistency(whetherexaminersgivethesameanswer)butratherofaccuracy(whethertheygivetherightanswer).Accuracycanbeevaluatedonlyfromlarge,appropriatelydesignedblack-boxstudies.


PCASTcouldfindnoblack-boxstudiesappropriatelydesignedtoestablishthefoundationalvalidityofidentificationsbasedonfootwearanalysis.

Consistentwithourconclusion,theOSACFootwearandTiresubcommitteerecentlyidentifiedtheneedforbothblack-boxandwhite-boxexaminerreliabilitystudies—citingitasa“majorgapincurrentknowledge”inwhichthereis“noorlimitedcurrentresearchbeingconducted.”340

Finding7:Footwearanalysis

Foundationalvalidity.PCASTfindstherearenoappropriateempiricalstudiestosupportthefoundationalvalidityoffootwearanalysistoassociateshoeprintswithparticularshoesbasedonspecificidentifyingmarks(sometimescalled“randomlyacquiredcharacteristics).Suchconclusionsareunsupportedbyanymeaningfulevidenceorestimatesoftheiraccuracyandthusarenotscientificallyvalid.

PCASThasnotevaluatedthefoundationalvalidityoffootwearanalysistoidentifyclasscharacteristics(forexample,shoesizeormake).

ThePathForward

Incontrasttolatentfingerprintanalysisandfirearmsanalysis,thereislittleresearchonwhichtobuildwithrespecttoconclusionsthatseektoassociateashoeprintwithaparticularshoe(identificationconclusions).

Newapproacheswillbeneededtodevelopparadigms.Asaninitialstep,theFBIisengaginginastudyexaminingasetof700similarbootsthatwerewornbyFBISpecialAgentcadetsduringtheir16-weektrainingprogram.ThestudyaimstoassesswhetherRACsareobservedonfootwearfromdifferentindividuals.Whilesuch“uniqueness”studies(i.e.,demonstrationsthatmanyobjectshavedistinctfeatures)cannotestablishfoundationalvalidity(seep.42),theimpressionsgeneratedfromthefootwearcouldprovideaninitialdatasetfor(1)apilotblack-boxstudyand(2)apilotdatabaseoffeaturefrequencies.Importantly,NISTisbeginninga

339Thepaperstatesthat“Allcharacteristicsandobservationsthatweretobeconsideredbytheexaminersduringthecomparisonswereclearlyidentifiedandlabeledforeachimpression.”340See:www.nist.gov/forensics/osac/upload/SAC-Phy-Footwear-Tire-Sub-R-D-001-Examiner-Reliability-Study_Revision_Feb_2016.pdf(accessedonMay,12,2016).


94

studytoseeifitispossibletoquantifythefootwearexaminationprocess,oratminimumaspectsoftheprocess,inanefforttoincreasetheobjectivityoffootwearanalysis.

Separately,evaluationsshouldbeundertakenconcerningtheaccuracyandreliabilityofdeterminationsaboutclasscharacteristics,atopicthatisnotaddressedinthisreport.

5.7HairAnalysisForensichairexaminationisaprocessbywhichexaminerscomparemicroscopicfeaturesofhairtodeterminewhetheraparticularpersonmaybethesourceofaquestionedhair.AsPCASTwascompletingthisreport,theDOJreleasedforcommentguidelinesconcerningtestimonyonhairexaminationthatincludedsupportingdocumentsaddressingthevalidityandreliabilityofthediscipline.341WhilePCASThasnotundertakenacomprehensivereviewofthediscipline,weundertookareviewofthesupportingdocumentinordertoshedfurtherlightonthestandardsforconductingascientificevaluationofaforensicfeature-comparisondiscipline.

Thesupportingdocumentstatesthat“microscopichaircomparisonhasbeendemonstratedtobeavalidandreliablescientificmethodology,”whilenotingthat“microscopichaircomparisonsalonecannotleadtopersonalidentificationanditiscrucialthatthislimitationbeconveyedbothinthewrittenreportandintestimony.”

FoundationalStudiesofMicroscopicHairExamination

Insupportofitsconclusionthathairexaminationisvalidandreliable,theDOJsupportingdocumentdiscussesonlyahandfulofstudiesofhumanhaircomparison.TheprimarysupportisaseriesofstudiesbyGaudettefrom1974,1976and1978.342The1974and1976studiesfocus,respectively,onheadhairandpubichair.Becausethedesignsandresultsaresimilar,wefocusontheheadhairstudy.

TheDOJsupportingdocumentstatesthat“Intheheadhairstudies,atotalof370,230intercomparisonswereconducted,withonlyninepairsofhairsthatcouldnotbedistinguished”(suggestingafalsepositiverateoflowerthan1in40,000).Morespecifically,thedesignofthis1974studywasasfollows:asingleexaminer(1)scoredbetween6and11headhairsfromeachof100individuals(atotalof861hairs)withrespectto23distinctcategories(withatotalof96possiblevalues);(2)comparedthehairsfromdifferentindividuals,toidentifythosepairsofhairswithfewerthanfourdifferences;and(3)comparedthesepairsofhairsmicroscopicallytoseeiftheycouldbedistinguished.

341See:DepartmentofJusticeProposedUniformLanguageforTestimonyandReportsfortheForensicHairExaminationDiscipline,availableat:https://www.justice.gov/dag/file/877736/downloadandSupportingDocumentationforDepartmentofJusticeProposedUniformLanguageforTestimonyandReportsfortheForensicHairExaminationDiscipline,availableat:https://www.justice.gov/dag/file/877741/download.342Gaudette,B.D.,Keeping,E.S.(1974).“Anattemptatdeterminingprobabilitiesinhumanscalphaircomparisons.”JournalofForensicSciences,19,599-606;Gaudette,B.D.(1976).“ProbabilitiesandHumanPubicHairComparisons.”JournalofForensicScience,21,514-517;Gaudette,B.D.(1978).“Somefurtherthoughtsonprobabilitiesandhumanhaircomparisons.”JournalofForensicSciences23,758–763.


95

TheDOJsupportingdocumentfailstonotethatthesestudieswerestronglycriticizedbyotherscientistsforflawedmethodology.343ThemostseriouscriticismwasthatGaudettecomparedonlyhairsfromdifferentindividuals,butdidnotlookathairsfromthesameindividual.Aspointedoutbya1990paperbytwoauthorsattheHairandFibreUnitoftheRoyalCanadianMountedPoliceForensicLaboratory,theapparentlylowfalsepositiveratecouldhaveresultedfrom(1)examinerbias—thatis,thattheexaminerexplicitlyknewthatallhairsbeingexaminedcamefromdifferentindividualsandthuscouldbeinclined,consciouslyorunconsciously,tosearchfordifferences,and(2)inconsistencyinidentifyingfeatures,whichwouldintroducerandomnoiseandtherebydecreasethelikelihoodofmatches.Inthe1990paper,theauthorsemployedasimilarstudydesign,butemployedtwoexaminerswhoexaminedallpairsofhairs.Theyfoundnon-repeatabilityfortheindividualexaminers(“eachexaminerhadconsiderableday-to-dayvariationinhairfeatureclassification”)andnon-reproducibilitybetweentheexaminers(“inmanycases,theexaminersclassifiedthesamehairsdifferently”).Mostnotably,theyfoundthat,whiletheexaminersfoundnomatchesbetweenhairsfromdifferentindividuals,theyalsofoundalmostnoconsistentmatchesamonghairsfromthesameperson.Of15pairsofsame-sourcehairsthattheauthorsdeterminedshouldhavebeendeclaredtomatch,onlytwowerecorrectlycalledbybothexaminers.

InGaudette’s1978study,theauthorgaveadifferenthairtoeachofthreeexaminertrainees,whohadcompletedoneyearoftraining,andaskedthemtoidentifyanymatchingsamplesamongareferencesetof100hairs(which,unbeknownsttotheexaminers,camefrom100differentpeople,includingthesourcesofthehairs).Thethreeexaminersreported1,1and4matches,consistingof3correctand3incorrectanswers.Ofthedeclaredmatches,50%werethusfalsepositiveassociations.Amongthe300totalcomparisons,theoverallfalsepositiveratewas1%,whichnotablyis400-foldhigherthantherateestimatedinthe1974study.

Interestingly,wenotedthattheDOJsupportingdocumentwronglyreportstheresultsofthestudy—claimingthatthethirdexaminertraineemadeonly1error,ratherthan3errors.Theexplanationforthisdiscrepancyisfoundinaremarkablyfrankpassageofthetext,whichillustratestheneedforemployingrigorousprotocolsinevaluatingtheresultsofexperiments:

“Twotraineescorrectlyidentifiedonehairandonlyonehairasbeingsimilartothestandard.Thethirdtraineefirstconcludedthattherewerefourhairssimilartothestandard.Uponcloserexaminationandconsultationwiththeotherexaminers,hewaseasilyabletoidentifyoneofhischoicesasbeingincorrect.However,hewasstillconvincedthattherewerethreehairssimilartothestandard,thecorrectoneandtwoothers.Examinationbytheauthorbroughttheopinionthatoneofthesetwootherscouldbeeliminatedbutthattheremainingonewasindistinguishablefromhairsinthestandard.Anotherexperiencedexaminerthenstudiedthe

343Wickenheiser,R.A.,Hepworth,D.G.(1990).Furtherevaluationofprobabilitiesinhumanscalphaircomparisons.JournalofForensicSciences,35,1323-1329.SeealsoBarnett,P.D.andOgle,R.R.(1982)Probabilitiesandhumanhaircomparison.JournalofForensicSciences.27,272–278.andGaudette,B.D.(1982)"ASupplementaryDiscussionofProbabilitiesandHumanHairComparisons,"JournalofForensicSciences,Vol.27,No.2,pp.279-289.


96

hairsandalsoconcludedthatoneofthetwootherscouldbeeliminated.Thistime,however,itwastheoppositetotheonepickedbytheauthor!”344

Expostfactoreclassificationoferrorsisgenerallynotadvisableinstudiespertainingtovalidityandreliability.Gaudette’smethodologyandresults—includinglowconsistencyandlowsensitivity—castdoubtonthestudiesasanappropriatebasisforestablishingfoundationalvalidityandestimatingthedegreeofreliability.345

FBIStudyComparingMicroscopicHairExaminationandDNAAnalysis

AnotherconcerningaspectoftheDOJsupportingdocumentisitstreatmentoftheFBIstudyonhairexaminationdiscussedinChapter2.Inthat2002study,FBIpersonnelusedmitochondrialDNAanalysistore-examine170samplesfrompreviouscasesinwhichtheFBILaboratoryhadperformedmicroscopichairexamination.Theauthorsfoundthat,in9of80cases(11%)inwhichtheFBILaboratoryhadfoundthehairstobemicroscopicallyindistinguishable,theDNAanalysisshowedthatthehairsactuallycamefromdifferentindividuals.

The2002FBIstudyisalandmarkinforensicsciencebecauseitwasthefirststudytosystematicallyandcomprehensivelyanalyzealargecollectionofpreviouscaseworktomeasurethefrequencyoffalse-positiveassociations.Itsconclusionisofenormousimportancetoforensicscience,topolice,tocourtsandtojuries:Whenhairexaminersconcludeincaseworkthattwohairsamplesaremicroscopicallyindistinguishable,thehairsoften(1in9times)comefromdifferentsources.

Surprisingly,theDOJdocumentcompletelyignoresthiskeyfinding.Instead,itreferencestheFBIstudyonlytosupportthepropositionthatDNAanalysis“canbeusedinconjunctionwithmicroscopichaircomparison,”citing“a2002study,whichindicatedthatoutof80microscopicassociations,approximately88%werealsoincludedbyadditionalmtDNAtesting.”Thedocumentfailstoacknowledgethattheremainingcaseswerefoundtobefalseassociations—thatis,misleadingresultsabouttheoriginsofthehairs.346

Conclusion

Ourbriefreviewisintendedsimplytoillustratepotentialpitfallsinevaluationsofthefoundationalvalidityandreliabilityofamethod.PCASTismindfuloftheconstraintsthatDOJfacesinundertakingscientificevaluationsofvalidityandreliabilityofforensicmethods,becausecriticalevaluationsbyDOJmightbetakenasadmissionsthatcouldbeusedtochallengepastconvictionsorpresentprosecutions. 344Gaudette,B.D.(1978).“Somefurtherthoughtsonprobabilitiesandhumanhaircomparisons.”JournalofForensicSciences23,758–763.345ThetwootherhumanhairstudiescitedintheDOJsupportingdocumentareStrauss,M.T.(1983).“Forensiccharacterizationofhumanhair.”TheMicroscope,31,15-29.andBisbing,R.E.,Wolner,M.F.(1984).“MicroscopicalDiscriminationofTwins’HeadHair.”JournalofForensicSciences,29,780-786.346Inafootnote,thedocumentalsotakespainstonotethatpapercannotbetakentoprovideanestimateofthefalse-positiverateformicroscopichaircomparison,becauseitcontainsnodataaboutthenumberofdifferent-sourcescomparisonthatexaminerscorrectlyexcluded.Whilethisstatementiscorrect,itismisleading—becausethepaperprovidesanestimateofafarmoreimportantquantity—namely,thefrequencyoffalseassociationsthatoccurredinactualcasework.


97

Theseissueshighlightwhyitisimportantforevaluationsofscientificvalidityandreliabilitytobecarriedoutbyascience-basedagencythatisnotitselfinvolvedintheapplicationofforensicsciencewithinthelegalsystem(seeSection6.1).

Theyalsounderscorewhyitisimportantthatquantitativeinformationaboutthereliabilityofmethods(e.g.,thefrequencyoffalseassociationsinhairanalysis)bestatedclearlyinexperttestimony.WereturntothispointinChapter8,whereweconsidertheDOJ’sproposedguidelines,whichwouldbarexaminersfromprovidinginformationaboutthestatisticalweightorprobabilityofaconclusionthataquestionedhaircomesfromaparticularsource.

5.8ApplicationtoadditionalmethodsAlthoughwehaveundertakendetailedevaluationsofonlyonsixspecificmethodsandincludedadiscussionofaseventhmethod,thebasicanalysiscanbeappliedtoassessthefoundationalvalidityofanyforensicfeature-comparisonmethod—includingtraditionalforensicdisciplinesaswellasmethodsyettobedeveloped(suchasmicrobiomeanalysisorinternet-browsingpatterns).

Wenotethattheevaluationofscientificvalidityisbasedontheavailablescientificevidenceatapointintime.Somemethodsthathavenotbeenshowntobefoundationallyvalidmayultimatelybefoundtobereliable—althoughsignificantmodificationstothemethodsmayberequiredtoachievethisgoal.Othermethodsmaynotbesalvageable—aswasthecasewithcompositionalbulletleadanalysisandislikelythecasewithbitemarks.Stillothersmaybesubsumedbydifferentbutmorereliablemethods,muchasDNAanalysishasreplacedothermethodsinmanyinstances.

5.9ConclusionAsthechapterabovemakesclear,manyforensicfeature-comparisonmethodshavehistoricallybeenassumedratherthanestablishedtobefoundationallyvalidbasedonappropriateempiricalevidence.Onlywithinthepastdecadehastheforensicsciencecommunitybeguntorecognizetheneedtoempiricallytestwhetherspecificmethodsmeetthescientificcriteriaforscientificvalidity.Onlyinthepastfiveyears,forexample,havetherebeenappropriatestudiesthatestablishthefoundationalvalidityandmeasurethereliabilityoflatentfingerprintanalysis.Formostsubjectivemethods,therearenoappropriateblack-boxstudieswiththeresultthatthereisnoappropriateevidenceoffoundationalvalidityorestimatesofreliability.

ThescientificanalysisandfindingsinChapters4and5areintendedtohelpfocustherelevantactorsonhowtoensurescientificvalidity,bothforexistingtechnologiesandfortechnologiesstilltobedeveloped.

PCASTexpectsthatsomeforensicfeature-comparisonmethodsmayberejectedbycourtsasinadmissiblebecausetheylackadequateevidenceofscientificvalidity.Wenotethatdecisionstoexcludeunreliablemethodshavehistoricallyhelpedpropelmajorimprovementsinforensicscience—ashappenedintheearlydaysofDNAevidence—withtheresultthatsomemethodsbecomeestablished(possiblyinrevisedform)asscientificallyvalid,whileothersarediscarded.


98

Intheremainingchapters,weofferrecommendationsonspecificactionsthatcouldbetakenbytheFederalGovernment—includingscience-basedagencies(NISTandOSTP),theFBILaboratory,theAttorneyGeneral,andtheFederaljudiciary—toensurethescientificvalidityandreliabilityofforensicfeature-comparisonmethodsandpromotetheirmorerigoroususeinthecourtroom.


99

6. ActionstoEnsureScientificValidityinForensicScience:RecommendationstoScience-basedAgencies

BasedonthescientificfindingsinChapters4and5,PCASThasidentifiedactionsthatwebelieveshouldbetakenbyscience-basedFederalagencies—specifically,NISTandOSTP—toensurethescientificvalidityofforensicfeature-comparisonmethods.

6.1RoleforNISTinongoingevaluationoffoundationalvalidityThereisanurgentneedforongoingevaluationofthefoundationalvalidityofimportantmethods,toprovideguidancetothecourts,theDOJ,andtheforensicsciencecommunity.Evaluationsshouldbeundertakenofbothexistingmethodologiesthathavenotyetmetthescientificstandardsforfoundationalvalidityandnewmethodologiesthatarebeingandwillbedevelopedintheyearsahead.Toensureunbiasedandscientificjudgments,suchevaluationsmustclearlybeoverseenbyanagencythatisscience-basedandisnotitselfinvolvedintheapplicationofforensicsciencewithinthelegalsystem.

ThisresponsibilityshouldbelodgedwithNIST.NISTistheworld’sleadingmetrologicallaboratory,withalonganddistinguishedhistoryinthescienceandtechnologyofmeasurement.Ithastremendousexperienceindesigningandcarryingoutvalidationstudies,aswellasassessingthefoundationalvalidityandreliabilityoflaboratorytechniquesandpractices.NIST’smissionofadvancingmeasurementscience,technology,andstandardshasexpandedfromtraditionalphysicalmeasurementstandardstorespondtomanyotherimportantsocietalneeds,includingthoseofforensicscience,inwhichNISThasvigorousprograms.347Asdescribedabove,NISThasbeguntoleadanumberofimportanteffortstostrengthentheforensicsciences,includingitsroleswithrespecttoNCFSandOSAC.

PCASTrecommendsthatNISTbetaskedwithresponsibilityforpreparinganannualreportevaluatingthefoundationalvalidityofkeyforensicfeature-comparisonmethods,basedonavailable,publishedempiricalstudies.TheseevaluationsshouldbeconductedundertheauspicesofNIST,withinputfromadditionalexpertiseasdeemednecessaryfromexpertsoutsideforensicscience,andoverseenbyanappropriatereviewpanel.Thereportsshould,asaminimum,produceassessmentsalongthelinesofthoseinthisreport,updatedasappropriate.OurintentionisnotthatNISThaveaformalregulatoryrolewithrespecttoforensicscience,butratherthatNIST’sevaluationshelpinformcourts,theDOJ,andtheforensicsciencecommunity.

WedonotexpectNISTtotakeresponsibilityforconductingthenecessaryvalidationstudies.However,NISTshouldadviseonthedesignandexecutionofsuchstudies.NISTcouldcarryoutsomestudiesthroughitsownintramuralresearchprogramandthroughCSAFE.However,themajorityofstudieswilllikelybeconductedby

347http://www.nist.gov/forensics.


100

othergroups—suchasNSF’splannedIndustry/UniversityCooperativeResearchCenters;theFBILaboratory;theU.S.nationallaboratories;otherFederalagencies;statelaboratories;andacademicresearchers.

WenotethattherecommendationsapprovedbytheNCFS’sSubcommitteeonScientificInquiryandcurrentlyunderconsiderationbythefullNCFSproposethatNISTtakeonaformalrolewithregardtoevaluatingthedevelopmentalvalidationofforensicsciencetestmethods.348Specifically,therecommendationsproposethatNISTestablishanin-houseentitywiththecapacitytoevaluateandassessthevalidationofforensicsciencemethods.

Finally,webelievethatthestateofforensicsciencewouldbeimprovedifpapersonthefoundationalvalidityofforensicfeature-comparisonmethodswerepublishedinleadingscientificjournalsratherthaninforensic-sciencejournals,where,owingtoweaknessesintheresearchcultureoftheforensicsciencecommunitydiscussedinthisreport,thestandardsforpeerreviewarelessrigorous.Commendably,FBIscientistspublisheditsblack-boxstudyoflatentfingerprintsintheProceedingsoftheNationalAcademyofSciences.WesuggestthatNISTexplorewithoneormoreleadingscientificjournalsthepossibilityofcreatingaprocessforrigorousreviewandonlinepublicationofimportantstudiesoffoundationalvalidityinforensicscience.AppropriatejournalscouldincludeMetrologia,aleadinginternationaljournalinpureandappliedmetrology,andtheProceedingsoftheNationalAcademyofSciences.

6.2AcceleratingthedevelopmentofobjectivemethodsAsdescribedthroughoutthereport,objectivemethodsaregenerallypreferabletosubjectivemethods.Thereasonsincludegreateraccuracy,greaterefficiency,lowerriskofhumanerror,lowerriskofcognitivebias,andgreatereaseofestablishingfoundationalvalidityandestimatingreliability.Wherepossible,vigorouseffortsshouldbeundertakentotransformsubjectivemethodsintoobjectivemethods.

Threeforensicfeature-comparisonmethods—DNAanalysisofcomplexmixtures,latentfingerprintanalysis,andfirearmsanalysis—areripeforsuchtransformation.Asdiscussedinthepreviouschapter,therearestrongreasonstobelievethat(1)DNAanalysisofcomplexmixturescanbemadeobjectivethroughappropriatealgorithmsthathavebeenopenlydisclosedandrigorouslytested,and(2)latentfingerprintanalysisandfirearmsanalysiscanbemadeobjectivethroughautomatedimageanalysis.

NIST,inconjunctionwiththeFBILaboratory,shouldplayaleadershiproleinpropellingthistransformationby(1)thecreationanddisseminationoflargedatasetstosupportthedevelopmentandtestingofmethodsbybothcompaniesandacademicresearchers,(2)grantandcontractsupport;and(3)sponsoringprocesses,suchasprizecompetitions,toevaluatemethods.

348NationalCommissiononForensicScience,“RequestforNISTtoEvaluateDevelopmentalValidationStudiesforForensicScienceTestMethodsinAdvanceofDocumentaryStandardsSetting,”approvedbySubcommitteeonScientificInquiryFebruary20,2016.SeealsoNationalCommissiononForensicScience,“ViewsoftheCommission:ValidationofForensicScienceMethodology,”ApprovedbySubcommitteeFebruary29,2016.Availableat:www.justice.gov/ncfs/file/831546/download.


101

6.3ImprovingOSACThecreationbyNISTofOSACwasanimportantstepinstrengtheningforensicsciencepractice.Theorganizationaldesign—whichhousesallofthesubjectareacommunitiesunderonestructureandencouragescross-disciplinarycommunicationandcoordination—isasignificantimprovementoverthepreviousScientificWorkingGroups(SWGs),whichfunctionedlessformallyasstand-alonecommittees.

However,initiallessonsfromitsfirstyearsofoperationhaverevealedsomeimportantshortcomings.OSAC’smembershipincludesrelativelyfewindependentscientists:itisdominatedbyforensicprofessionals,whomakeupmorethantwo-thirdsofitsmembers.Similarly,ithasfewindependentstatisticians:whilevirtuallyallofthestandardsandguidelinesevaluatedbythisbodyneedconsiderationofstatisticalprinciples,OSAC’s600membersincludeonly14statisticiansspreadacrossall4SACsand23subcommittees.

Restructuring

PCASTconcludesthatOSAClackssufficientindependentscientificexpertiseandoversighttoovercometheseriousflawsinforensicscience.Somerestructuringisnecessarytoensurethatindependentscientistsandstatisticianshaveagreatervoiceinthestandardsdevelopmentprocess,arequirementformeaningfulscientificvalidity.Mostimportantly,OSACshouldhaveaformalcommittee—aMetrologyResourceCommittee—attheleveloftheotherthreeResourceCommittees(theLegalResourceCommittee,theHumanFactorsCommittee,andtheQualityInfrastructureCommittee).ThisCommitteeshouldbecomposedoflaboratoryscientistsandstatisticiansfromoutsidetheforensicsciencecommunityandchargedwithreviewingeachstandardandguidelinethatisrecommendedforregistryapprovalbytheScienceAreaCommitteesbeforeitissentforfinalreviewtheForensicScienceStandardsBoard(FSSB).

AvailabilityofOSACstandards

OSACisnotaformalstandard-settingbody.ItreviewsandevaluatesstandardsrelevanttoforensicsciencedevelopedbystandardsdevelopingorganizationssuchasASTMInternational,theNationalFireProtectionAssociation(NFPA)andtheInternationalOrganizationforStandardization(ISO)forinclusionontheOSACRegistriesofStandardsandGuidelines.TheOSACevaluationprocessincludesapubliccommentperiod.OSAC,workingwiththestandardsdevelopers,hasarrangedforthecontentofstandardsunderconsiderationtobeaccessibletothepublicduringthepubliccommentperiod.OnceapprovedbyOSAC,astandardislisted,bytitle,onapublicregistrymaintainedbyNIST.Itiscustomaryforsomestandardsdevelopingorganization,includingASTMInternational,tochargeafeeforalicensedcopyofeachcopyrightedstandardandtorestrictusersfromdistributingthesestandards.349,350

NISTrecentlynegotiatedalicensingagreementwithASTMInternationalthat,forafee,allowsfederal,stateandlocalgovernmentemployeesonlineaccesstoASTMCommitteeE30standards.351However,thislistdoesnot

349ForalistofASTM’sforensicsciencestandards,see:www.astm.org/DIGITAL_LIBRARY/COMMIT/PAGES/E30.htm.350TheAmericanAcademyofForensicSciences(AAFS)willalsobecomeanaccreditedSDOandcould,inthefuture,developstandardsforreviewandlistingbyOSAC.351Accordingtotherevisedcontract,ASTMwillprovideunlimitedweb-basedaccessforallASTMcommitteeE30Forensic


102

includeindigentdefendants,privatedefenseattorneys,orlargeswathsoftheacademicresearchcommunity.Atpresent,contractshavebeennegotiatedwiththeotherSDOsthathavestandardscurrentlyunderreviewbytheOSAC.PCASTbelievesitisimportantthatstandardsintendedforuseinthecriminaljusticesystemarewidelyavailabletoallwhomayneedaccess.Itisimportantthatthestandardsbereadilyavailabletodefendantsandtoexternalobservers,whohaveanimportantroletoplayinensuringqualityincriminaljustice.352

NISTshouldensurethatthecontentofOSAC-registeredstandardsandguidelinesarefreelyavailabletoanypartythatmaydesiretheminconnectionwithalegalcaseorforevaluationandresearch,includingbyaligningwiththepoliciesrelatedtoreasonableavailabilityofstandardsintheOfficeofManagementandBudgetCircularA-119,FederalParticipationintheDevelopmentandUseofVoluntaryConsensusStandardsandConformityAssessmentActivitiesandtheOfficeoftheFederalRegister,IBR(incorporationbyreference)Handbook.

6.4NeedforanR&DstrategyforForensicScienceThe2009NRCreportfoundthatthereisanurgentneedtostrengthenforensicscience,notingthat,“Forensicscienceresearchisnotwellsupported,andthereisnounifiedstrategyfordevelopingaforensicscienceresearchplanacrossfederalagencies.”353

Itisespeciallyimportanttocreateandsupportavibrantacademicresearchcommunityrootedinthescientificcultureofuniversities.Thiswillrequiresignificantfundingtosupportacademicresearchgroups,butwillpaybigdividendsindrivingqualityandinnovationinbothexistingandentirelynewmethods.

BothNISTandNSFhaverecentlytakeninitialstepstohelpbridgethesignificantgapsbetweentheforensicpractitionerandacademicresearchcommunitiesthroughmulti-disciplinaryresearchcenters.Thesecenterspromisetoengagethebroaderresearchcommunityinadvancingforensicscienceandcreateneededlinksbetweentheforensicsciencecommunityandabroadbaseofresearchuniversitiesandcouldhelpdriveforwardcriticalfoundationalresearch.

Nonetheless,asnotedinChapter2,thetotallevelofFederalfundingbyNIJ,NIST,andNSFtotheacademiccommunityforfundamentalresearchinforensicscienceisextremelysmall.Substantiallylargerfundingwillbeneededtodeveloparobustresearchcommunityandtosupportthedevelopmentandevaluationofpromisingnewtechnologies.

FederalR&Deffortsinforensicscience,bothintramuralandextramural,needtobebettercoordinated.Nooneagencyhasleadresponsibilityforensuringthattheforensicsciencesareadequatelysupported.GreatercoordinationisneededacrosstherelevantFederalagenciesandlaboratoriestoensurethatfundingisdirectedtothehighestprioritiesandthatworkisofhighquality.

ScienceStandardsto:OSACmembersandaffiliates;NISTandFederal/State/LocalCrimeLaboratories;PublicDefendersOffices;LawEnforcementAgencies;ProsecutorOffices;andMedicalExaminer/andCoronersOffices.352PCASTexpressesnoopinionabouttheappropriatenessofpaywallsforstandardsinareasotherthancriminaljustice.353NationalResearchCouncil.StrengtheningForensicScienceintheUnitedStates:APathForward.TheNationalAcademiesPress.WashingtonDC.(2009):p.78.


103

OSTPshouldconvenerelevantFederalagencies,laboratories,andstakeholderstodevelopanationalresearchstrategyand5-yearplantoensurethatfoundationalresearchinsupportoftheforensicsciencesiswell-coordinated,solidifyFederalagencycommitmentsmadetodate,andgalvanizefurtheractionandfundingthatcouldbetakentoencourageadditionalfoundationalresearch,improvecurrentforensicmethods,supportthecreationofnewresearchdatabases,andoverseetheregularreviewandprioritizationofresearch.

6.5Recommendations

Basedonitsscientificfindings,PCASTmakesthefollowingrecommendations.

Recommendation1.Assessmentoffoundationalvalidity

Itisimportantthatongoingscientificevaluationsofthefoundationalvaliditybeconductedtoassess,onanongoingbasis,currentandnewlydevelopedforensicfeature-matchingtechnologies.Toensureunbiasedscientificjudgments,suchevaluationsshouldbeconductedbyanagencythatisscience-basedandisnotitselfinvolvedintheapplicationofforensicsciencewithinthelegalsystem.

(a)TheNationalInstituteofStandardsandTechnology(NIST)shouldperformsuchevaluationsandshouldissueanannualpublicreportevaluatingthefoundationalvalidityofkeyforensicfeature-comparisonmethods.

(i)Theevaluationsshould(a)assesswhethereachmethodreviewedhasbeenadequatelydefinedandwhetheritsfoundationalvalidityhasbeenadequatelyestablishedbyempiricalevidence;(b)bebasedonstudiespublishedinthescientificliteraturebythelaboratoriesandagenciesintheU.S.andinothercountries,aswellasanyworkconductedbyNIST’sownstaffandgrantees;(c)asaminimum,produceassessmentsalongthelinesofthoseinthisreport,updatedasappropriate;and(d)beconductedundertheauspicesofNIST,withadditionalexpertiseasdeemednecessaryfromexpertsoutsideforensicscience.

(ii)NISTshouldestablishanadvisorycommitteeofexperimentalandstatisticalscientistsfromoutsidetheforensicsciencecommunitytoprovideadviceconcerningtheevaluationsandtoensurethattheyarerigorousandindependent.ThemembersoftheadvisorycommitteeshouldbeselectedjointlybyNISTandtheOfficeofScienceandTechnologyPolicy.

(iii)NISTshouldprioritizeforensicfeature-comparisonmethodsthataremostinneedofevaluation,includingthosecurrentlyinuseandinlate-stagedevelopment,basedoninputfromtheDepartmentofJusticeandthescientificcommunity.

(iv)WhereNISTassessesthatamethodhasbeenestablishedasfoundationallyvalid,itshould(a)indicateappropriateestimatesoferrorratesbasedonfoundationalstudies,and(b)identifyanyissuesrelevanttovalidityasapplied.

(v)WhereNISTassessesthatamethodhasnotbeenestablishedasfoundationallyvalid,itshould


104

suggestwhatsteps,ifany,couldbetakentoestablishthemethod’svalidity.

(vi)NISTshouldnothaveregulatoryresponsibilitieswithrespecttoforensicscience.

(vii)NISTshouldencourageoneormoreleadingscientificjournalsoutsidetheforensiccommunitytodevelopmechanismstopromotetherigorouspeerreviewandpublicationofpapersaddressingthefoundationalvalidityofforensicfeature-comparisonmethods.

(b)ThePresidentshouldrequestandCongressshouldprovideincreasedappropriationstoNISTof(a)$4milliontosupporttheevaluationactivitiesdescribedaboveand(b)$10milliontosupportincreasedresearchactivitiesinforensicscience,includingoncomplexDNAmixtures,latentfingerprints,voice/speakerrecognition,andface/irisbiometrics.

Recommendation2.DevelopmentofobjectivemethodsforDNAanalysisofcomplexmixturesamples,latentfingerprintanalysis,andfirearmsanalysis

(a)TheNationalInstituteofStandardsandTechnology(NIST)shouldtakealeadershiproleintransformingthreeimportantfeature-comparisonmethods—DNAanalysisofcomplexmixtures,latentfingerprintanalysisandfirearmsanalysis—fromcurrentlysubjectivemethodsintoobjectivemethods.

(i)NISTshouldcoordinatetheseeffortswiththeFederalBureauofInvestigationLaboratory,theDefenseForensicScienceCenter,theNationalInstituteofJustice,andotherrelevantagencies.

(ii)Theseeffortsshouldinclude(1)thecreationanddisseminationoflargedatasetsandtestmaterials(suchascomplexDNAmixtures)tosupportthedevelopmentandtestingofmethodsbybothcompaniesandacademicresearchers,(2)grantandcontractsupport;and(3)sponsoringprocesses,suchasprizecompetitions,toevaluatemethods.

Recommendation3.ImprovingtheOrganizationforScientificAreaCommitteesprocess

(a)TheNationalInstituteofStandardsandTechnology(NIST)shouldimprovetheOrganizationforScientificAreaCommittees(OSAC),whichwasestablishedtodevelopandpromulgatestandardsandguidelinestoimprovebestpracticesintheforensicsciencecommunity.

(i)NISTshouldestablishaMetrologyResourceCommittee,composedofmetrologists,statisticians,andotherscientistsfromoutsidetheforensicsciencecommunity.ArepresentativeoftheMetrologyResourceCommitteeshouldserveoneachoftheScientificAreaCommittees(SACs)toprovidedirectguidanceontheapplicationofmeasurementandstatisticalprinciplestothedevelopingdocumentarystandards.


105

(ii)TheMetrologyResourceCommittee,asawhole,shouldreviewandpublicallyapproveordisapproveallstandardsproposedbytheScientificAreaCommitteesbeforetheyaretransmittedtotheForensicScienceStandardsBoard.

(b)NISTshouldensurethatthecontentofOSAC-registeredstandardsandguidelinesarefreelyavailabletoanypartythatmaydesiretheminconnectionwithalegalcaseorforevaluationandresearch,includingbyaligningwiththepoliciesrelatedtoreasonableavailabilityofstandardsintheOfficeofManagementandBudgetCircularA-119,FederalParticipationintheDevelopmentandUseofVoluntaryConsensusStandardsandConformityAssessmentActivitiesandtheOfficeoftheFederalRegister,IBR(incorporationbyreference)Handbook.

Recommendation4.R&Dstrategyforforensicscience

(a)TheOfficeofScienceandTechnologyPolicy(OSTP)shouldcoordinatethecreationofanationalforensicscienceresearchanddevelopmentstrategy.Thestrategyshouldaddressplansandfundingneedsfor:

(i)majorexpansionandstrengtheningoftheacademicresearchcommunityworkingonforensicsciences,includingsubstantiallyincreasedfundingforbothresearchandtraining;

(ii)studiesoffoundationalvalidityofforensicfeature-comparisonmethods;

(iii)improvementofcurrentforensicmethods,includingconvertingsubjectivemethodsintoobjectivemethods,anddevelopmentofnewforensicmethods;

(iv)developmentofforensicfeaturedatabases,withadequateprivacyprotections,thatcanbeusedinresearch;

(v)bridgingthegapbetweenresearchscientistsandforensicpractitioners;and

(vi)oversightandregularreviewofforensicscienceresearch.

(b)Inpreparingthestrategy,OSTPshouldseekinputfromappropriateFederalagencies,includingespeciallytheDepartmentofJustice,DepartmentofDefense,NationalScienceFoundation,andNationalInstituteofStandardsandTechnology;federalandstateforensicsciencepractitioners;forensicscienceandnon-forensicscienceresearchers;andotherstakeholders.


106

7. ActionstoEnsureScientificValidityinForensicScience:RecommendationstotheFBILaboratory

BasedonthescientificfindingsinChapters4and5,PCASThasidentifiedactionsthatwebelieveshouldbetakenbytheFBILaboratorytoensurethescientificvalidityofforensicfeature-comparisonmethods.

WenotethattheFBILaboratoryhasplayedanimportantroleinrecentyearsinundertakinghigh-qualityscientificstudiesoflatentfingerprintanalysis.PCASTapplaudstheseeffortsandurgestheFBILaboratorytoexpandthem.

7.1RoleforFBILaboratoryTheFBILaboratoryisafull-service,state-of-the-artfacilitythatworkstoapplycutting-edgesciencetosolvecasesandpreventcrime.Itsmissionistoapplyscientificcapabilitiesandtechnicalservicestothecollection,processing,andexploitationofevidencefortheLaboratoryandotherdulyconstitutedlawenforcementandintelligenceagenciesinsupportofinvestigativeandintelligencepriorities.Currently,theLaboratoryemploysapproximately750employeesandover300contractorstomeetthebroadscopeofthismission.

Laboratorycapabilitiesandservices

TheFBIhasspecializedcapabilitiesandpersonneltorespondtoincidents,collectevidenceintheirfield,carryoutforensicanalyses,andprovideexpertwitnesstestimony.TheFBILaboratorysupportsEvidenceResponseTeamsinall56FBIfieldofficesandhaspersonnelwhospecializeinhazardousevidenceandcrimescenedocumentationanddatacollection.TheLaboratoryisresponsiblefortrainingandsupplyingtheseresponseactivitiesforFBIpersonnelacrosstheU.S.354TheLaboratoryalsomanagestheTerroristExplosiveDeviceAnalyticalCenter(TEDAC),whichreceivednearly1,000evidencesubmissionsinFY2015anddisseminatedover2,000intelligenceproducts.

TheFBILaboratoryemploysforensicexaminerstocarryoutanalysesinarangeofdisciplines,includingchemistry,cryptanalysis,DNA,firearmsandtoolmarks,latentprints,questioneddocuments,andtraceevidence.TheFBILaboratoryreceivedover3875evidencesubmissionsandauthoredover4850laboratoryreportsinFY

354TheFBILaboratorysupported162deploymentsand168responseexercises,aswellasdelivering239trainingcoursesinFY2015.


107

2015.Inadditiontocarryingoutcaseworkforfederalcases,theLaboratoryprovidessupporttostateandlocallaboratoriesandcarriesouttestinginstateandlocalcasesforsomedisciplines.

Researchanddevelopmentactivities

Inadditiontoitsservices,theFBILaboratorycarriesoutimportantresearchanddevelopmentactivities.TheactivitiesarecriticalforprovidingtheLaboratorywiththemostadvancedtoolsforadvancingitsmission.AstrongresearchprogramandcultureisalsoimportanttotheLaboratory’sabilitytomaintainexcellenceandtoattractandretainhighlyqualifiedpersonnel.

Duetotheexpansivescopeandmanyrequirementsonitsoperations,onlyaboutfivepercentoftheFBILaboratory’sannual$100millionbudgetisavailableforresearchanddevelopmentactivities.355TheR&Dbudgetisstretchedacrossanumberofappliedresearchactivities,includingvalidationstudies(fornewmethodsorcommercialproducts,suchasnewDNAanalyzers).Foritsinternalresearchactivities,theLaboratoryreliesheavilyonitsVisitingScientistProgram,whichbringsapproximately25postdocs,master’sstudents,andbachelor’sdegreestudentsintothelaboratoryeachyear.TheLaboratoryhasworkedtopartnerwithothergovernmentagenciestoprovidemoreresourcestoitsresearchprioritiesasacompositeinitiative,andhasalsobeenabletostretchavailablebudgetsbyperformingcriticalresearchstudiesincrementallyoverseveralyears.

TheFBILaboratory’sseriesofstudiesinlatentprintexaminationisanexampleofimportantfoundationalresearchthatitwasabletocarryoutincrementallyoverafive-yearperiod.Theworkincludes“blackbox”studiesthatevaluatetheaccuracyandreliabilityoflatentprintexaminers’conclusions,aswellas“whitebox”studiestoevaluatehowthequalityandquantityoffeaturesrelatetolatentprintexaminers’decisions.Thesestudieshaveresultedinaseriesofimportantpublicationsthathavehelpedtoquantifyerrorratesforthecommunityofpracticeandassesstherepeatabilityandreproducibilityoflatentfingerprintexaminers’decisions.Indeed,PCAST’sjudgmentthatlatentfingerprintanalysisisfoundationallyvalidrestsheavilyontheFBIblack-boxstudy.Similarlinesofresearcharebeingpursuedinsomeotherdisciplines,includingfirearmsexaminationandquestioneddocuments.

Unfortunately,thelimitedfundingavailableforthesestudies—andfortheintramuralresearchprogrammoregenerally—hashamperedprogressintestingthefoundationalvalidityofforensicsciencemethodsandinstrengtheningtheforensicsciences.PCASTbelievesthatthebudgetfortheFBILaboratoryshouldbesignificantlyincreased,andtargetedsoasallowtheR&Dbudgettobeincreasedtoatotalof$20million.

Accesstodatabases

TheFBIalsohasanimportantroletoplayinencouragingresearchbyexternalscientists,byfacilitatingaccess,underappropriateconditions,tolargeforensicdatabases.Mostofthedatabasesroutinelyusedinforensicanalysisarenotaccessibleforusebyresearchers,andthelackofaccesshampersprogressinimprovingforensic

355In2014,theFBILaboratoryspent$10.9milliononforensicscienceresearchanddevelopment,withroughlyhalffromitsownbudgetandhalffromgrantsfromNISTandtheDepartmentofHomelandSecurity.See:NationalAcademiesofSciences,Engineering,andMedicine.SupportforForensicScienceResearch:ImprovingtheScientificRoleoftheNationalInstituteofJustice.TheNationalAcademiesPress.WashingtonDC.(2015):p.31.


108

science.Forexample,ballisticdatabasesystemssuchastheBureauofAlcohol,Tobacco,FirearmsandExplosives’NationalIntegratedBallisticInformationSystem(NIBIN),whichissearchedbyfirearmsexaminersseekingtoidentifyafirearmorcartridgecase,cannotbeassessedtostudyitscompleteness,relevanceorquality,andthesearchalgorithmthatisusedtoidentifypotentialmatchescannotbeevaluated.TheNGI(formerlyIAFIS)356systemthatcurrentlyhousesmorethan70millionfingerprintentrieswoulddramaticallyexpandthedataavailableforstudy;currently,thereexistsonlyonepubliclyavailablefingerprintdatabase,consistingof258latentprint-10printpairs.357And,theFBI’sCODISsystem,whichcurrentlyhousesmorethan14millionoffenderandarresteeDNAprofiles.NISThasdevelopedaninventoryofalloftheforensicdatabasesthatareheavilyusedbylawenforcementandforensicscientists,withinformationastotheiraccessibility.

Substantialeffortsareneededtomakeexistingforensicdatabasesmoreaccessibletotheresearchcommunity,subjecttoappropriateprotectionofprivacy,suchasremovalofpersonallyidentifiableinformationanddata-userestrictions.

Notably,thelawthatauthorizestheFBItomaintainanationalforensicDNAdatabaseexplicitlycontemplatesallowingaccesstoDNAsamplesandDNAanalyses“ifpersonallyidentifiableinformationisremoved...foridentificationresearchandprotocoldevelopmentpurposes.”358Inhumangenetics,whichposessimilarorgreaterprivacyconcerns,appropriatedataaccesspolicieshavebeendevelopedtoencouragebroaderengagementbyacademicresearchcommunitytoDNAdatabases.359SuchpoliciesshouldbefeasibleforforensicDNAdatabasesaswell.Forlatentprints,privacyconcernsmightbeamelioratedinvarietyofways.Forexample,onemightavoidtheissueby(1)generatinglargecollectionsofknown-latentprintpairswithvaryingqualityandquantityofinformationthroughthetouchingandhandlingofnaturalitemsinawidevarietyofcircumstances(surfaces,pressure,distortion,etc.);(2)usingsoftwaretoautomaticallygeneratethe“morphingtransformations”fromtheknownprintsandthelatentprints;and(3)applyingthesetransformationstoprintsfromdeceasedindividuals.360

Forotherdisciplinessuchasfirearmsanalysisandtreadmarks,therearenosignificantprivacyconcerns.

PCASTbelievesthattheavailabilityofdatawillspeedthedevelopmentofmethods,tools,andsoftwarethatwillimproveforensicscience.Fordatabasesunderitscontrol,theFBILaboratoryshoulddevelopprogramstomake 356NGIstandardsfor“NextGenerationIdentification”andcombinesmultiplebiometricinformationsystems,includingIAFIS,irisandfacerecognitionsystems,andothers. 357NISTSpecialDatabase27,availableat:www.nist.gov/srd/nistsd27.cfm.358FederalDNAIdentificationAct,42U.S.C.§14132(b)(3)(D)).359Anumberofmodelsthathavebeendevelopedinthebiomedicalresearchcontextthatallowfortieredaccesstosensitivedatawhileprovidingadequateprivacyprotectioncouldbeemployedhere.ResearcherscouldberequiredtosignNon-DisclosureAgreements(NDAs)orenterintolimiteduseagreements.Researcherscouldberequiredtoaccessthedataonsite,sothatdatacannotbedownloadedorshared,orcouldbepermittedtodownloadonlyaggregatedorsummarydata.360Medicalexaminersofficesroutinelycollectfingerprintsfromdeceasedindividualsaspartoftheautopsyprocess;thesefingerprintscouldbecollectedandusedtocreatealargedatabaseforresearchpurposes.


109

forensicdatabases(orsubsetsofthosedatabases)accessibletoresearchersunderconditionsthatprotectprivacy.Fordatabasesownedbyothers,theFBILaboratoryandNISTshouldeachworkwithotheragenciesandcompaniesthatcontrolthedatabasestodevelopprogramsprovidingappropriateaccess.

7.2RecommendationsBasedonitsscientificfindings,PCASTmakesthefollowingrecommendations.

Recommendation5.ExpandedForensic-ScienceAgendaattheFederalBureauofInvestigationLaboratory

(a)Researchprograms.TheFederalBureauofInvestigation(FBI)Laboratoryshouldundertakeavigorousresearchprogramtoimproveforensicscience,buildingonitsrecentimportantworkonlatentfingerprintanalysis.Theprogramshouldinclude:

(i)conductingstudiesonthereliabilityoffeature-comparisonmethods,inconjunctionwithindependentthirdpartieswithoutastakeintheoutcome;

(ii)developingnewapproachestoimprovereliabilityoffeature-comparisonmethods;

(iii)expandingcollaborativeprogramswithexternalscientists;and

(iv)ensuringthatexternalscientistshaveappropriateaccesstodatasetsandsamplecollections,sothattheycancarryoutindependentstudies;and

(b)Black-boxstudies.Drawingonitsexpertiseinforensicscienceresearch,theFBILaboratoryshouldassistinthedesignandexecutionofadditionalblack-boxstudiesforsubjectivemethods,includingforlatentfingerprintanalysisandfirearmsanalysis.Thesestudiesshouldbeconductedbyorinconjunctionwithindependentthirdpartieswithnostakeintheoutcome.

(c)Developmentofobjectivemethods.TheFBILaboratoryshouldworktotransformthreeimportantfeature-comparisonmethods—DNAanalysisofcomplexmixtures,latentfingerprintanalysisandfirearmsanalysis—fromcurrentlysubjectivemethodsintoobjectivemethods.Theseeffortsshouldinclude(i)thecreationanddisseminationoflargedatasetstosupportthedevelopmentandtestingofmethodsbybothcompaniesandacademicresearchers,(ii)grantandcontractsupport;and(iii)sponsoringprizecompetitionstoevaluatemethods.

(c)Proficiencytesting.TheFBILaboratory,shouldpromoteincreasedrigorinproficiencytestingby(i)withinthenextfouryears,institutingroutineblindproficiencytestingwithintheflowofcaseworkinitsownlaboratory,(ii)assistingotherFederal,Stateandlocallaboratoriesindoingsoaswell;and(iii)encouragingroutineaccesstoandevaluationofthetestsusedincommercialproficiencytesting.

(e)Latentfingerprintanalysis.TheFBILaboratoryshouldvigorouslypromotetheadoption,byall


110

laboratoriesthatperformlatentfingerprintanalysis,ofrulesrequiringa“linearACE”process—wherebyexaminersmustcompleteanddocumenttheiranalysisofalatentfingerprintbeforelookingatanyknownfingerprintandshouldseparatelydocumentanyadditionaldatausedduringcomparisonandevaluation.

(f)Transparencyconcerningqualityissuesincasework.TheFBILaboratory,aswellasotherFederalforensiclaboratories,shouldregularlyandpubliclyreportqualityissuesincasework(similartothepracticesemployedbytheNetherlandsForensicInstitute,describedinChapter5),asameanstoimprovequalityandpromotetransparency.

(g)Budget.ThePresidentshouldrequestandCongressshouldprovideincreasedappropriationstotheFBItorestoretheFBILaboratory’sbudgetforforensicscienceresearchactivitiesfromitscurrentlevelto$30millionandshouldevaluatetheneedforincreasedfundingforotherforensic-scienceresearchactivitiesintheDepartmentofJustice.


111

8. ActionstoEnsureScientificValidityinForensicScience:RecommendationstotheAttorneyGeneral

BasedonthescientificfindingsinChapters4and5,PCASThasidentifiedactionsthatwebelieveshouldbetakenbytheAttorneyGeneraltoensurethescientificvalidityofforensicfeature-comparisonmethodsandpromotetheirmorerigoroususeinthecourtroom.

8.1EnsuringtheuseofscientificallyvalidmethodsinprosecutionsTheFederalGovernmenthasadeepcommitmenttoensuringthatcriminalprosecutionsarenotonlyfairintheirprocess,butcorrectintheiroutcome—thatis,thatguiltyindividualsareconvicted,whileinnocentindividualsarenot.

Towardthisend,theDOJshouldensurethattestimonyaboutforensicevidencepresentedincourtisbasedonmethodsthatarevalidfoundationallyandasapplied.ThisreportprovidesguidanceforDOJconcerningthescientificcriteriaforvalidityingeneral,aswellasevaluationsofsixspecificforensicmethods.Overthelongterm,DOJshouldlooktoongoingevaluationsofforensicmethodsthatshouldbeperformedbyNIST(asdescribedinChapter6).

Intheinterim,DOJshouldundertakeareviewofforensicfeature-comparisonmethods(beyondthosereviewedinthisreport)toidentifywhichmethodsusedbyDOJlackappropriateblack-boxstudiesnecessarytoassessfoundationalvalidity.Becausesuchsubjectivemethodsarepresumptivelynotestablishedtobefoundationallyvalid,DOJshouldevaluate(1)whetherDOJshouldpresentincourtconclusionsbasedonsuchmethodsand(2)whetherblack-boxstudiesshouldbelaunchedtoevaluatethosemethods.

8.2RevisionofDOJRecentlyProposedGuidelinesonExpertTestimonyOnJune3,2016,theDOJreleasedforcommentafirstsetofproposedguidelines,togetherwithsupportingdocuments,on“ProposedUniformLanguageforTestimonyandReports”onseveralforensicsciences,includinglatentfingerprintanalysisandforensicfootwearandtireimpressionanalysis.361OnJuly21,2016,theDOJreleasedforcommentasecondsetofproposedguidelinesandsupportingdocumentsforseveraladditionalforensicsciences,includingmicroscopichairanalysis,certaintypesofDNAanalysis,andotherfields.

361See:www.justice.gov/dag/proposed-language-regarding-expert-testimony-and-lab-reports-forensic-science.AsecondsetofproposedguidelineswasreleasedonJuly21,2016includinghairanalysisandmitochondrialDNAandYchromosometyping(www.justice.gov/dag/proposed-uniform-language-documents-anthropology-explosive-chemistry-explosive-devices-geology).


112

Theguidelinesrepresentanimportantstepforward,becausetheyinstructDOJexaminersnottomakesweepingclaimsthattheycanidentifythesourceofafingerprintorfootprinttotheexclusionofallotherpossiblesources.PCASTapplaudsDOJ’sintentionandeffortstobringuniformityandtopreventinaccuratetestimonyconcerningfeaturecomparisons.

Someaspectsoftheguidelines,however,arenotscientificallyappropriateandembodyheterodoxviewsofthekinddiscussedinSection4.7.Asanillustration,wefocusontheguidelinesforfootwearandtireimpressionanalysisandtheguidelinesforhairanalysis.

Footwearandtireimpressionanalysis

RelevantportionsoftheguidelinesfortestimonyandreportsaboutforensicfootwearandtireimpressionareshowninBox6.

BOX6.ExcerptfromDOJProposeduniformlanguagefortestimonyandreportsfortheforensicfootwearandtireimpressiondiscipline362

StatementsApprovedforUseinLaboratoryReportsandExpertWitnessTestimonyRegardingForensicExaminationofFootwearandTireImpressionEvidence

Identification

1.Theexaminermaystatethatitishis/heropinionthattheshoe/tireisthesourceoftheimpressionbecausethereissufficientqualityandquantityofcorrespondingfeaturessuchthattheexaminerwouldnotexpecttofindthatsamecombinationoffeaturesrepeatedinanothersource.Thisisthehighestdegreeofassociationbetweenaquestionedimpressionandaknownsource.Thisopinionrequiresthatthequestionedimpressionandtheknownsourcecorrespondinclasscharacteristicsandalsoshareoneormorerandomlyacquiredcharacteristics.Thisopinionacknowledgesthatanidentificationtotheexclusionofallotherscanneverbeempiricallyproven.

StatementsNotApprovedforUseinLaboratoryReportsandExpertWitnessTestimonyRegardingForensicExaminationofFootwearandTireImpressionEvidence

ExclusionofAllofOthers

1.Theexaminermaynotstatethatashoe/tireisthesourceofaquestionedimpressiontotheexclusionofallothershoes/tiresbecauseallothershoes/tireshavenotbeenexamined.Examiningalloftheshoes/tiresintheworldisapracticalimpossibility.

ErrorRate

2.Theexaminermaynotstateanumericalvalueorpercentageregardingtheerrorrateassociatedwitheitherthemethodologyusedtoconducttheexaminationsortheexaminerwhoconductedtheanalyses.

362See:www.justice.gov/olp/file/861936/download.


113

StatisticalWeight

3.Theexaminermaynotstateanumericalvalueorprobabilityassociatedwithhis/heropinion.Accurateandreliabledataand/orstatisticalmodelsdonotcurrentlyexistformakingquantitativedeterminationsregardingtheforensicexaminationoffootwear/tireimpressionevidence.

Theseproposedguidelineshaveseriousproblems.

Anexaminermayopinethatashoeisthesourceofanimpression,butnotthattheshoeisthesourceofimpressiontotheexclusionofallotherpossibleshoes.But,asamatteroflogic,thereisnodifferencebetweenthesetwostatements.IfanexaminerbelievesthatXisthesourceofY,thenheorshenecessarilybelievesthatnothingelseisthesourceofY.Anysensiblejurorshouldunderstandthisequivalence.

Whatthenisthegoaloftheguidelines?Itappearstobetoacknowledgethepossibilityoferror.Ineffect,examinersshouldsay,“IbelieveXisthesourceofY,althoughIcouldbewrongaboutthat.”

Thisisappropriate.But,thecriticalquestionisthen:Howlikelyisitthattheexamineriswrong?

There’stherub:theguidelinesbartheexaminerfromdiscussingthelikelihoodoferror,becausethereisnoaccurateorreliableinformationaboutaccuracy.Ineffect,examinersareinstructedtosay,“IbelieveXisthesourceofY,althoughIcouldbewrongaboutthat.But,IhavenoideahowoftenI’mwrongbecausewehavenoreliableinformationaboutthat.”

Suchastatementdoesnotmeetanyplausibletestofscientificvalidity.AsJudgeEasterlywroteinWilliamsv.UnitedStates,aclaimofidentificationundersuchcircumstances:

hasthesameprobativevalueasthevisionofapsychic:itreflectsnothingmorethantheindividual’sfoundationlessfaithinwhathebelievestobetrue.Thisisnotevidenceonwhichwecaningoodconsciencerely,particularlyincriminalcases,wherewedemandproof—realproof—beyondareasonabledoubt,preciselybecausethestakesaresohigh.363

Hairanalysis

RelevantportionsoftheguidelinesfortestimonyandreportsonforensichairexaminationareshowninBox7.

BOX7.ExcerptfromDOJProposeduniformlanguagefortestimonyandreportsfortheforensichairexaminationdiscipline364

StatementsNotApprovedforUseinForensicHairExaminationTestimonyand/orLaboratoryReports

363Williamsv.UnitedStates,DCCourtofAppeals,DecidedJanuary21,2016,(Easterly,concurring).364DepartmentofJusticeProposedUniformLanguageforTestimonyandReportsfortheForensicHairExaminationDiscipline,availableat:https://www.justice.gov/dag/file/877736/download


114

Comparisons

HumanHair

1.Theexaminermaystateorimplythatthequestionedhumanhairismicroscopicallyconsistentwiththeknownhairsampleandaccordingly,thesourceoftheknownhairsamplecanbeincludedasapossiblesourceofthequestionedhair.

StatementsNotApprovedforUseinForensicHairExaminationTestimonyand/orLaboratoryReports

Individualization

1.Theexaminermaynotstateorimplythatahaircamefromaparticularsourcetotheexclusionofallothers.

StatisticalWeight

2.Theexaminermaynotstateorimplyastatisticalweightorprobabilitytoaconclusionorprovidealikelihoodthatthequestionedhairoriginatedfromaparticularsource.determinationsregardingtheforensicexaminationoffootwear/tireimpressionevidence.

ZeroErrorRate

3.Theexaminermaynotstateorimplythatthemethodusedinperformingmicroscopichairexaminationshasazeroerrorrateorisinfallible.

Theguidelinesappropriatelystatethatexaminersmaynotclaimthattheycanindividualizethesourceofahairnorthattheyhaveazeroerrorrate.However,whileexaminersmay“stateorimplythatthequestionedhumanhairismicroscopicallyconsistentwiththeknownhairsampleandaccordingly,thesourceoftheknownhairsamplecanbeincludedasapossiblesourceofthequestionedhair,”theyarebarredfromprovidingaccurateinformationaboutthereliabilityofsuchconclusions.Thisiscontrarytothescientificrequirementthatforensicfeature-comparisonmethodsmustbesupportedbyandaccompaniedbyappropriateempiricalestimatesofreliability.

Inparticular,asdiscussedinSection5.7,alandmarkstudyin2002byscientistsattheFBILaboratoryshowedthat,among80instancesinactualcaseworkwhereexaminersconcludedthataquestionedhairwasmicroscopicallyconsistentwiththeknownhairsample,thehairwerefoundbyDNAanalysistohavecomefromadifferentsourcein11%ofcases.Thefactthatsuchasignificantproportionofconclusionswerefalseassociationsisoftremendousimportancetointerpretingconclusionsofhairexaminers.

IncasesofhairexaminationunaccompaniedbyDNAanalysis,examinersshouldberequiredtodisclosethehighfrequencyoffalseassociationsseenintheFBIstudysothatjuriescanappropriatelyweighconclusions.


115

Conclusion

TheDOJshouldrevisetheproposedguidelines,tobringthemintoalignmentwithstandardsforscientificvalidity.Thesupportingdocumentationshouldalsoberevised,asdiscussedinSection5.7.


Recommendation6.Useoffeature-comparisonmethodsinFederalprosecutions

(a)TheAttorneyGeneralshoulddirectattorneysappearingonbehalfoftheDepartmentofJustice(DOJ)toensureexperttestimonyincourtaboutforensicfeature-comparisonmethodsmeetsthestandardsofscientificvalidity.

Whilepretrialinvestigationsmaydrawonawiderrangeofmethods,experttestimonyincourtaboutforensicfeature-comparisonmethodsincriminalcases—whichcanbehighlyinfluentialandhasledtomanywrongfulconvictions—mustmeetahigherstandard.Inparticular,attorneysappearingonbehalfoftheDOJshouldensurethat:

(i)theforensicfeature-comparisonmethodsonwhichtestimonyisbasedhavebeenestablishedtobefoundationallyvalid,asshownbyappropriateempiricalstudiesandconsistencywithevaluationsbytheNationalInstituteofStandardsandTechnology,whereavailable;and

(ii)thetestimonyisscientificallyvalid,withtheexpert’sstatementsconcerningtheaccuracyofmethodsandtheprobativevalueofproposedidentificationsbeingconstrainedbytheempiricallysupportedevidenceandnotimplyingahigherdegreeofcertainty.

(b)DOJshouldundertakeaninitialreview,withassistancefromtheNationalInstituteofStandardsandTechnology,ofsubjectivefeature-comparisonmethodsusedbyDOJtoidentifywhichmethods(beyondthosereviewedinthisreport)lackappropriateblack-boxstudiesnecessarytoassessfoundationalvalidity.Becausesuchsubjectivemethodsarepresumptivelynotestablishedtobefoundationallyvalid,DOJshouldevaluatewhetheritisappropriatetopresentincourtconclusionsbasedonsuchmethods.

(c)Whererelevantmethodshavenotyetbeenestablishedtobefoundationallyvalid,DOJshouldencourageandprovidesupportforappropriateblack-boxstudiestoassessfoundationalvalidityandmeasurereliability.Thedesignandexecutionofthesestudiesshouldbeconductedbyorinconjunctionwithindependentthirdpartieswithnostakeintheoutcome.

Recommendation7.DepartmentofJusticeguidelinesonexperttestimony

(a)TheAttorneyGeneralshouldreviseandreissueforpubliccommenttheDepartmentofJustice’s(DOJ)proposed“UniformLanguageforTestimonyandReports”andsupportingdocumentstobring


116

themintoalignmentwithstandardsforscientificvalidity.

(b)TheAttorneyGeneralshouldissueinstructionsdirectingthat:

(i)Whereempiricalstudiesand/orstatisticalmodelsexisttoshedlightontheaccuracyofaforensicfeature-comparisonmethod,anexaminershouldprovidequantitativeinformationabouterrorrates,inaccordancewithguidelinestobeestablishedbyDOJandtheNationalInstituteofStandardsandTechnology,basedonadvicefromthescientificcommunity.

(ii)Wheretherearenotadequateempiricalstudiesand/orstatisticalmodelstoprovidemeaningfulinformationabouttheaccuracyofaforensicfeature-comparisonmethod,DOJattorneysandexaminersshouldnotoffertestimonybasedonthemethod.Ifitisnecessarytoprovidetestimonyconcerningthemethod,theyshouldclearlyacknowledgetocourtsthelackofsuchevidence.

(iii)Intestimony,examinersshouldalwaysstateclearlythaterrorscananddooccur,duebothtosimilaritiesbetweenfeaturesandtohumanmistakesinthelaboratory.


117

9. ActionstoEnsureScientificValidityinForensicScience:

RecommendationstotheJudiciary

BasedonthescientificfindingsinChapters4and5,PCASThasidentifiedactionsthatwebelieveshouldbetakenbythejudiciarytoensurethescientificvalidityofevidencebasedonforensicfeature-comparisonmethodsandpromotetheirmorerigoroususeinthecourtroom.

9.1ScientificvalidityasafoundationforadmissibilityofexperttestimonyInFederalcourts,judgesareassignedthecriticalroleof“gatekeepers”chargedwithensuringthereliabilityofexperttestimony.365Specifically,Rule702(c,d)oftheFederalRulesofEvidencerequiresthat(1)experttestimonymustbetheproductof“reliableprinciplesandmethods”and(2)expertsmusthave“reliablyapplied”themethodstothefactsofthecase”.366TheSupremeCourthasheldthatjudgesmustdetermine“whetherthereasoningormethodologyunderlyingthetestimonyisscientificallyvalid.”367

AsdiscussedinChapter3,thisframeworkestablishesanimportantconversationbetweenthejudiciaryandthescientificcommunity.Theadmissibilityofexperttestimonydependsonathresholdtestofwhetheritmeetscertainlegalstandardsforreliability,whichareexclusivelytheprovinceofthejudiciary.Yet,theselegalstandardsaretobe“baseduponscientificvalidity”.368

PCASTdoesnotopineonthelegalstandards,butaimsinthisreporttoclarifythescientificstandardsthatunderliethem.Toensurethatthedistinctionbetweenscientificandlegalconceptsisclear,wehaveadoptedspecifictermstorefertoscientificconcepts(foundationalvalidityandvalidityasapplied)intendedtoparallellegalconceptsexpressedinRule702(c,d).

AstheSupremeCourthasnoted,thejudge’sinquiryunderRule702isaflexibleone;thereisnosimpleone-size-fits-alltestthatcanbeapplieduniformlytoallscientificdisciplines.369Rather,theevaluationofscientificvalidityshouldbebasedontheappropriatescientificcriteriaforthescientificfield.Moreover,theappropriatescientificfieldshouldbethelargerscientificdisciplinetowhichitbelongs.370

365Daubertv.MerrellDowPharmaceuticals,509U.S.579(1993).366See:www.uscourts.gov/file/rules-evidence.www.uscourts.gov/file/rules-evidence.367Daubertv.MerrellDowPharmaceuticals,509U.S.579(1993)at592.368Daubert,atFN9(“inacaseinvolvingscientificevidence,evidentiaryreliabilitywillbebasedonscientificvalidity.”)369Daubert,at594.370Forexample,inFrye,thecourtevaluatedwhetheraprofferedliedetectorhadgained“standingandscientificrecognitionamongphysiologicalandpsychologicalauthorities,”ratherthanamongliedetectorexperts.Fryev.United


118

Inthisreport,PCASThasfocusedonforensicfeature-comparisonmethods—whichbelongtothefieldofmetrology,thescienceofmeasurementanditsapplication.371Wehavesought—inaformusablebycourts,aswellasbyscientistsandotherswhoseektoimproveforensicscience—tolayoutthescientificcriteriaforfoundationalvalidityandvalidityasapplied(Chapter4)andtoillustratetheirapplicationtospecificforensicfeature-comparisonmethods(Chapter5).

ThescientificcriteriaaredescribedinFinding1.Theymightbesummarizedasfollows:

Scientificvalidityandreliabilityrequirethatamethodhasbeensubjectedtoempiricaltesting,underconditionsappropriatetoitsintendeduse,thatprovidesvalidestimatesofhowoftenthemethodreachesanincorrectconclusion.Forsubjectivefeature-comparisonmethods,appropriatelydesignedblack-boxstudiesarerequired,inwhichmanyexaminersrenderdecisionsaboutmanyindependenttests(typically,involving“questioned”samplesandoneormore“known”samples)andtheerrorratesaredetermined.Withoutappropriateestimatesofaccuracy,anexaminer’sstatementthattwosamplesaresimilar—orevenindistinguishable—isscientificallymeaningless:ithasnoprobativevalue,andconsiderablepotentialforprejudicialimpact.Nothing—notpersonalexperiencenorprofessionalpractices—cansubstituteforadequateempiricaldemonstrationofaccuracy.

Theapplicationstospecificfeature-comparisonmethodsaredescribedinFindings2-7.ThefullsetofscientificfindingsiscollectedinChapter10.

9.2RoleofpastprecedentOneimportantissuethatarosethroughoutourdeliberationswastheroleofpastprecedents.

AsdiscussedinChapter5,ourscientificreviewfoundthatmostforensicfeature-comparisonmethods(withthenotableexceptionofDNAanalysisofsingle-sourceandsimple-mixturesamples)havehistoricallybeenassumedratherthanestablishedtobefoundationallyvalid.Onlyafteritbecameclearinrecentyears(basedonDNAandotheranalysis)thattherearefundamentalproblemswiththereliabilityofsomeofthesemethodshastheforensicsciencecommunitybeguntorecognizetheneedtoempiricallytestwhetherspecificmethodsmeetthescientificcriteriaforscientificvalidity.

Thiscreatesanobvioustension,becausemanycourtsadmitforensicfeature-comparisonmethodsbasedonlongstandingprecedentsthatweresetbeforethesefundamentalproblemswerediscovered.

Fromapurelyscientificstandpoint,theresolutionisclear.Whennewfactsfalsifyoldassumptions,courtsshouldnotbeobligedtodefertopastprecedents:theyshouldlookafreshatthescientificissues.

States,293F.1013(D.C.Cir.1923).Similarly,thefactthatbitemarkexaminersbelievethatbitemarkexaminationisvalidcarrieslittleweight.371Seefootnote89onp.25.


119

WesoughtadvicefromourpanelofSeniorAdvisorstounderstandhowsuchtensionsareresolvedwithinthelegalsystem.PCASTwasadvisedthattheSupremeCourthasmadeclearthatacourtmayoverruleprecedentifitfindsthatanearliercasewas“erroneouslydecidedandthatsubsequenteventshaveundermineditscontinuingvalidity.”373

PCASTexpressesnoviewonthelegalquestionofwhetheranypastcaseswere“erroneouslydecided.”However,PCASTnotesthat,fromascientificstandpoint,subsequenteventshaveindeedunderminedthecontinuingvalidityofconclusionsthatwerenotbasedonappropriateempiricalevidence.Theseeventsinclude(1)therecognitionofsystemicproblemswithsomeforensicfeature-comparisonmethods,includingthroughstudyofthecausesofhundredsofwrongfulconvictionsrevealedthroughDNAandotheranalysis;(2)the2009NRCreportfromtheNationalAcademyofSciences,theleadingexternalscientificadvisorybodyestablishedbytheLegislativeBranch,374thatfoundthatsomeforensicfeature-comparisonmethodslackascientificfoundation;and(3)thescientificreviewinthisreportbyPCAST,theleadingexternalscientificadvisorybodyestablishedbytheExecutiveBranch,375findingthatsomeforensicfeature-comparisonmethodslackfoundationalvalidity.

9.3ResourcesforjudgesAnotherimportantissuethatarosefrequentlyinourconversationswithexpertswastheneedforbetterresourcesforjudgesrelatedtoevaluationofforensicfeature-comparisonmethodsforuseinthecourts.

ThemostappropriatebodiestoprovidesuchresourcesaretheJudicialConferenceoftheUnitedStatesandtheFederalJudicialCenter.

TheJudicialConferenceoftheUnitedStatesisthenationalpolicy-makingbodyforthefederalcourts.376Itsstatutoryresponsibilityincludesstudyingtheoperationandeffectofthegeneralrulesofpracticeandprocedureinthefederalcourts.TheJudicialConferencedevelopsbestpracticesmanualsandissuesAdvisoryCommitteenotestoassistjudgeswithrespecttospecifictopics,includingthroughitsStandingAdvisoryCommitteeontheFederalRulesofEvidence.

TheFederalJudicialCenteristheresearchandeducationagencyofthefederaljudicialsystem.377Itsstatutorydutiesinclude(1)conductingandpromotingresearchonfederaljudicialproceduresandcourtoperations,and

373BoysMarkets,Inc.v.RetailsClerksUnion,398U.S.235,238(1970).Seealso:Pattersonv.McLeanCreditUnion,485U.S.617,618(1988)(notingthattheCourthas“overruledstatutoryprecedentsinahostofcases”).374TheNationalAcademyofScienceswascharteredbyCongressin1863toadvisetheFederalgovernmentonmattersofscience(U.S.Code,Section36,Title1503).375ThePresidentformallyestablishedastandingscientificadvisorycouncilsoonafterthelaunchofSputnikin1957.ItiscurrentlytitledthePresident’sCouncilofAdvisorsofScienceandTechnology(operatingunderExecutiveOrder13539,asamendedbyExecutiveOrder13596).376Createdin1922underthenametheConferenceofSeniorCircuitJudges,theJudicialConferenceoftheUnitedStatesiscurrentlyestablishedunder28U.S.C.§331.377TheFederalJudicialCenterwasestablishedbyCongressin1967(28U.S.C.§§620-629),ontherecommendationoftheJudicialConferenceoftheUnitedStates.


120

(2)conductingandpromotingorientationandcontinuingeducationandtrainingforfederaljudges,courtemployees,andothers.

PCASTrecommendsthattheJudicialConferenceoftheUnitedStates,throughitsSubcommitteeontheFederalRulesofEvidence,developbestpracticesmanualsandanAdvisoryCommitteenoteandtheFederalJudicialCenterdevelopeducationalprogramsrelatedtoproceduresforevaluatingthescientificvalidityofforensicfeature-comparisonmethods.


Recommendation8.Scientificvalidityasafoundationforadmissibilityofexperttestimony

(a)Whendecidingtheadmissibilityofexperttestimony,Federaljudgesshouldtakeintoaccounttheappropriatescientificcriteriaforassessingscientificvalidityincluding:

(i)foundationalvalidity,withrespecttotherequirementunderRule702(c)thattestimonyistheproductofreliableprinciplesandmethods;and

(ii)validityasapplied,withrespecttorequirementunderRule702(d)thatanexperthasreliablyappliedtheprinciplesandmethodstothefactsofthecase.

ThesescientificcriteriaaredescribedinFinding1.

(b)Federaljudges,whenpermittinganexperttotestifyaboutafoundationallyvalidfeature-comparisonmethod,shouldensurethattestimonyabouttheaccuracyofthemethodandtheprobativevalueofproposedidentificationsisscientificallyvalidinthatitislimitedtowhattheempiricalevidencesupports.Statementssuggestingorimplyinggreatercertaintyarenotscientificallyvalidandshouldnotbepermitted.Inparticular,courtsshouldneverpermitscientificallyindefensibleclaimssuchas:‘zero’,‘vanishinglysmall,’‘essentiallyzero,’‘negligible,’‘minimal,’or‘microscopic’errorrates;‘100%certainty’orproof‘toareasonabledegreeofscientificcertainty’;identification‘totheexclusionofallothersources;’orachanceoferrorsoremoteastobea‘practicalimpossibility.’

(c)Toassistjudges,theJudicialConferenceoftheUnitedStates,throughitsStandingAdvisoryCommitteeontheFederalRulesofEvidence,shouldprepare,withadvicefromthescientificcommunity,abestpracticesmanualandanAdvisoryCommitteenote,providingguidancetoFederaljudgesconcerningtheadmissibilityunderRule702ofexperttestimonybasedonforensicfeature-comparisonmethods.

(d)Toassistjudges,theFederalJudicialCentershoulddevelopprogramsconcerningthescientificcriteriaforscientificvalidityofforensicfeature-comparisonmethods.


121

10. ScientificFindingsPCAST’sscientificfindingsinthisreportarecollectedbelow.Finding1,concerningthescientificcriteriaforscientificvalidity,isbasedonthediscussioninChapter4.Findings2-6,concerningfoundationalvalidityofsixforensicfeature-comparisonmethods,isbasedontheevaluationsinChapter5.

Finding1:ScientificCriteriaforScientificValidityofaForensicFeature-ComparisonMethod

(1)Foundationalvalidity.Toestablishfoundationalvalidityforaforensicfeature-comparisonmethod,thefollowingelementsarerequired:

(a)areproducibleandconsistentprocedurefor(i)identifyingfeatureswithinevidencesamples;(ii)comparingthefeaturesintwosamples;and(iii)determining,basedonthesimilaritybetweenthefeaturesintwosamples,whetherthesamplesshouldbedeclaredtobelikelytocomefromthesamesource(“matchingrule”);and

(b)empiricalestimates,fromappropriatelydesignedstudiesfrommultiplegroups,thatestablish(i)themethod’sfalsepositiverate—thatis,theprobabilityitdeclaresaproposedidentificationbetweensamplesthatactuallycomefromdifferentsources,and(ii)themethod’ssensitivity—thatis,theprobabilityitdeclaresaproposedidentificationbetweensamplesthatactuallycomefromthesamesource.

AsdescribedinBox4,scientificvalidationstudiesshouldsatisfyanumberofcriteria:(a)Theyshouldbebasedonsufficientlylargecollectionsofknownandrepresentativesamplesfromrelevantpopulations;(b)theyshouldbeconductedsothathavenoinformationaboutthecorrectanswer;(c)thestudydesignandanalysisplanarespecifiedinadvanceandnotmodifiedafterwardsbasedontheresults;(d)thestudyisconductedoroverseenbyindividualsororganizationswithnostakeintheoutcome;(e)data,softwareandresultsshouldbeavailabletoallowotherscientiststoreviewtheconclusions;and(f)toensurethattheresultsarerobustandreproducible,thereshouldbemultipleindependentstudiesbyseparategroupsreachingsimilarconclusions.

Onceamethodhasbeenestablishedasfoundationallyvalidbasedonadequateempiricalstudies,claimsaboutthemethod’saccuracyandtheprobativevalueofproposedidentifications,inordertobevalid,mustbebasedonsuchempiricalstudies.

Forobjectivemethods,foundationalvaliditycanbeestablishedbydemonstratingthereliabilityofeachof


122

theindividualsteps(featureidentification,featurecomparison,matchingrule,falsematchprobability,andsensitivity).

Forsubjectivemethods,foundationalvaliditycanbeestablishedonlythroughblack-boxstudiesthatmeasurehowoftenmanyexaminersreachaccurateconclusionsacrossmanyfeature-comparisonproblemsinvolvingsamplesrepresentativeoftheintendeduse.Intheabsenceofsuchstudies,asubjectivefeature-comparisonmethodcannotbeconsideredscientificallyvalid.

Foundationalvalidityisasinequanon,whichcanonlybeshownthroughempiricalstudies.Importantly,goodprofessionalpractices—suchastheexistenceofprofessionalsocieties,certificationprograms,accreditationprograms,peer-reviewedarticles,standardizedprotocols,proficiencytesting,andcodesofethics—cannotsubstituteforempiricalevidenceofscientificvalidityandreliability.

(2)Validityasapplied.Onceaforensicfeature-comparisonmethodhasbeenestablishedasfoundationallyvalid,itisnecessarytoestablishitsvalidityasappliedinagivencase.

AsdescribedinBox5,validityasappliedrequiresthat:(a)Theforensicexaminermusthavebeenshowntobecapableofreliablyapplyingthemethod,asshownbyappropriateproficiencytesting(seeSection4.6),andmustactuallyhavedoneso,asdemonstratedbytheproceduresactuallyusedinthecase,theresultsobtained,andthelaboratorynotes,whichshouldbemadeavailableforscientificreviewbyothers;and(b)Theforensicexaminer’sassertionsabouttheprobativevalueofproposedidentificationsmustbescientificallyvalid—includingthattheexpertshouldreporttheoverallfalsepositiverateandsensitivityforthemethodestablishedinthestudiesoffoundationalvalidity;demonstratethatthesamplesusedinthefoundationalstudiesarerelevanttothefactsofthecase;whereapplicable,reportprobativevalueoftheobservedmatchbasedonthespecificfeaturesobservedinthecase;andnotmakeclaimsorimplicationsthatgobeyondtheempiricalevidence.

Finding2:DNAAnalysis

Foundationalvalidity.PCASTfindsthatDNAanalysisofsingle-sourcesamplesorsimplemixturesoftwoindividuals,suchasfrommanyrapekits,isanobjectivemethodthathasbeenestablishedtobefoundationallyvalid.

Validityasapplied.Becauseerrorsduetohumanfailureswilldominatethechanceofcoincidentalmatches,thescientificcriteriaforvalidityasappliedrequirethatanexpert(1)shouldhaveundergonerigorousandrelevantproficiencytestingtodemonstratetheirabilitytoreliablyapplythemethod,(2)shouldroutinelydiscloseinreportsandtestimonywhether,whenperformingtheexamination,heorshewasawareofanyfactsofthecasethatmightinfluencetheconclusion,and(3)shoulddisclose,uponrequest,allinformationaboutqualitytestingandqualityissuesinhisorherlaboratory.

Finding3:DNAanalysisofcomplex-mixturesamples


123

Foundationalvalidity.PCASTfindsthat:

(1)SubjectiveanalysisofcomplexDNAmixtures,includingwiththewidely-usedCombined-Probability-of-Inclusionmethods,hasnotbeenestablishedtobefoundationallyvalid.

(2)ObjectiveanalysisofcomplexDNAmixtureswithprobabilisticgenotypingsoftwareisrelativelynewandpromisingapproach.Beforethemethodcanbeestablishedasfoundationallyvalidforabroadrangeofsettings,moreresearchisrequiredappropriatelytoestablishthecapabilitiesandlimitationsofvariousapproaches.Atpresent,publishedpaperssupportthefoundationalvalidityofanalysis,withsomeprograms,ofDNAmixturesof3individualsinwhichthecontributorinquestionconstitutesatleast20%oftheintactDNAinthemixture.

Finding4:Bitemarkanalysis

Foundationalvalidity.PCASTfindsthatbitemarkanalysisdoesnotmeetthescientificstandardsforfoundationalvalidity,andisfarfrommeetingsuchstandards.Tothecontrary,availablescientificevidencestronglysuggeststhatexaminerscannotconsistentlyagreeonwhetheraninjuryisahumanbitemarkandcannotidentifythesourceofbitemarkwithreasonableaccuracy.

Finding5:Latentfingerprintanalysis

Foundationalvalidity.Basedlargelyonatworecentappropriatelydesignedblack-boxstudies,PCASTfindsthatlatentfingerprintanalysisisafoundationallyvalidsubjectivemethodology—albeitwithafalsepositiveratethatissubstantialandislikelytobehigherthanexpectedbymanyjurorsbasedonlongstandingclaimsabouttheinfallibilityoffingerprintanalysis.

Conclusionsofaproposedidentificationmaybescientificallyvalid,providedthattheyareaccompaniedbyaccurateinformationaboutlimitationsonthereliabilityoftheconclusion—specifically,that(1)onlytwoproperlydesignedstudiesofthefoundationalvalidityandaccuracyoflatentfingerprintanalysishavebeenconducted,(2)thesestudiesfoundfalsepositiveratesthatcouldbeashighas1errorin306casesinonestudyand1errorin18casesintheother,and(3)becausetheexaminerswereawaretheywerebeingtested,theactualfalsepositiverateincaseworkmaybehigher.Atpresent,claimsofhigheraccuracyarenotwarrantedorscientificallyjustified.Additionalblack-boxstudiesareneededtoclarifythereliabilityofthemethod.

Validityasapplied.Althoughweconcludethatthemethodisfoundationallyvalid,thereareanumberofopenissuesrelatedtoitsvalidityasapplied.

(1)Confirmationbias.WorkbyFBIscientistshasshownthatexaminerstypicallyalterthefeaturesthattheyinitiallymarkinalatentprintbasedoncomparisonwithanapparentlymatchingexemplar.Suchcircularreasoningintroducesaseriousriskofconfirmationbias.Examinersshouldberequired


124

tocompleteanddocumenttheiranalysisofalatentfingerprintbeforelookingatanyknownfingerprintandshouldseparatelydocumentanyadditionaldatausedduringtheircomparisonandevaluation.

(2)Contextualbias.Workbyacademicscholarshasshownthatexaminers’judgmentscanbeinfluencedbyirrelevantinformationaboutthefactsofacase.Effortsshouldbemadetoensurethatexaminersarenotexposedtopotentiallybiasinginformation.

(3)Proficiencytesting.Proficiencytestingisessentialfordetermininganexaminer’sabilitytomakeaccuratejudgments.Asdiscussedelsewhereinthisreport,proficiencytestingneedstobeimprovedbymakingitmorerigorous,byincorporatingitwithintheflowofcaseworkmorerigorous,andbydisclosingtestsforevaluationbythescientificcommunity.

Fromascientificstandpoint,validityasappliedrequiresthatanexpert:(1)hasundergonerelevantproficiencytestingtotesthisorheraccuracyandreportstheresultsoftheproficiencytesting;(2)discloseswhetherheorshedocumentedthefeaturesinthelatentprintinwritingbeforecomparingittotheknownprint;(3)providesawrittenanalysisexplainingtheselectionandcomparisonofthefeatures;(4)discloseswhether,whenperformingtheexamination,heorshewasawareofanyotherfactsofthecasethatmightinfluencetheconclusion;and(5)verifiesthatthelatentprintinthecaseathandissimilarinqualitytotherangeoflatentprintsconsideredinthefoundationalstudies.

Finding6:Firearmsanalysis

Foundationalvalidity.PCASTfindsthatfirearmsanalysiscurrentlyfallsshortofthecriteriaforfoundationalvalidity,becausethereisonlyasingleappropriatelydesignedstudytomeasurevalidityandestimatereliability.Thescientificcriteriaforfoundationalvalidityrequiremorethanonesuchstudy,todemonstratereproducibility.

Whetherfirearmsanalysisshouldbedeemedadmissiblebasedoncurrentevidenceisadecisionthatbelongstothecourts.

Iffirearmsanalysisisallowedincourt,thescientificcriteriaforvalidityasappliedshouldbeunderstoodtorequireclearlyreportingtheerrorratesseeninappropriatelydesignedblack-boxstudies(estimatedat1in66,witha95%confidencelimitof1in46,intheonesuchstudytodate).

Validityasapplied.Iffirearmsanalysisisallowedincourt,validityasappliedwould,fromascientificstandpoint,requirethattheexpert:

(1)hasundergonerigorousproficiencytestingonalargenumberoftestproblemstomeasurehisorheraccuracyanddisclosestheresultsoftheproficiencytesting;and

(2)discloseswhether,whenperformingtheexamination,heorshewasawareofanyotherfactsof


125

thecasethatmightinfluencetheconclusion.

Finding7:Footwearanalysis

Foundationalvalidity.PCASTfindstherearenoappropriateempiricalstudiestosupportthefoundationalvalidityoffootwearanalysistoassociateshoeprintswithparticularshoesbasedonspecificidentifyingmarks(sometimescalled“randomlyacquiredcharacteristics).Suchconclusionsareunsupportedbyanymeaningfulevidenceorestimatesoftheiraccuracyandthusarenotscientificallyvalid.

PCASThasnotevaluatedthefoundationalvalidityoffootwearanalysistoidentifyclasscharacteristics(forexample,shoesizeormake).


126

AppendixA:StatisticalIssuesToenhanceitsaccessibilitytoabroadaudience,themaintextofthisreportavoids,wherepossible,theuseofmathematicalandstatisticalterminology.However,fortheactualimplementationofsomeoftheprinciplesstatedinthereport,somewhatmoreprecisedescriptionsarenecessary.ThisAppendixsummarizestherelevantconceptsfromelementarystatistics.378

Sensitivityandfalsepositiverate

Forensicfeature-comparisonmethodstypicallyaimtodeterminehowlikelyitisthattwosamplescamefromthesamesource,giventheresultofaforensictestonthesamples.Twopossibilitiesareconsidered:thenullhypothesis(H0)thattheyarefromdifferentsources(H0)andthealternativehypothesis(H1)thattwosamplesarefromthesamesource.Theforensictestresultmaybesummarizedasmatchdeclared(M)ornomatchdeclared(O).

Therearetwonecessarycharacterizationsofamethod’saccuracy:Sensitivity(abbreviatedSEN)andFalsePositiveRate(FPR).

Sensitivityisdefinedastheprobabilitythatthemethoddeclaresamatchbetweentwosampleswhentheyareknowntobefromthesamesource(drawnfromanappropriatepopulation),thatis,SEN=P(M|H1).Forexample,avalueSEN=0.95wouldindicatethattwosamplesfromthesamesourcewillbedeclaredasamatch95%ofthetime.Inthestatisticsliterature,SENissometimesalsocalledthe“truepositiverate,”“TPR,”or“recallrate.”379

Falsepositiverate(abbreviatedFPR)isdefinedastheprobabilitythatthemethoddeclaresamatchbetweentwosamplesthatarefromdifferentsources(againinanappropriatepopulation),thatis,FPR=P(M|H0).Forexample,avalueFPR=0.01wouldindicatethattwosamplesfromdifferentsourceswillbe(mistakenly)calledasamatch1%ofthetime.380MethodswithahighFPRarescientificallyunreliableformakingimportant

378See,e.g.:PeterAmitage,G.Berry,JNSMatthews:StatisticalMethodsinMedicalResearch,4thed.,BlackwellScience,2002.;GeorgeSnedecor,WilliamGCochran:StatisticalMethods,8thed.,IowaStateUniversityPress,1989;GeraldvanBelle,LloydDFisher,PatrickHeagerty,ThomasLumley,Biostatistics:AMethodologyfortheHealthSciences,Wiley,2004.;AlanAgresti;BrentA.Coull:ApproximateIsBetterthan"Exact"forIntervalEstimationofBinomialProportions.TheAmericanStatistician52(2),119-126,1998.;RobertVHogg,ElliotTanis,DaleZimmerman:ProbabilityandStatisticalInference,9thed.,Pearson,2015.;DavidFreedman,RogerPisani,RogerPurves:Statistics.Norton,2007.;LincolnEMoses:ThinkandExplainwithStatistics,Addison-Wesley,1986.;DavidSMoore,GeorgePMcCabe,BruceACraig:IntroductiontothePracticeofStatistics.W.H.Freeman,2009.379ThetermfalsenegativerateissometimesusedforthecomplementofSEN,thatis,FNR=1–SEN.380Statisticiansmayrefertoamethod’sspecificity(SPC)insteadofitsfalsepositiverate(FPR).ThetwoarerelatedbytheformulaFPR=1–SPC.Intheexamplegiven,FPR=0.01(1%)andSPC=0.99(99%).


127

judgmentsincourtaboutthesourceofasample.Tobeconsideredreliable,theFPRshouldcertainlybelessthan5%anditmaybeappropriatethatitbeconsiderablylower,dependingontheintendedapplication.

Theresultsofagivenempiricalstudycanbesummarizedbyfourvalues:thenumberofoccurrencesinthestudyoftruepositives(TP),falsepositives(FP),falsenegatives(FN),andtruenegatives(TN).(Thematrixofthesevaluesis,perhapsoddly,referredtoasthe“confusionmatrix”.)

TestResult

Match NoMatch

H1:Trulyfromsamesource TP FN

H0:Trulyfromdifferentsources FP TN

Inthisstandard-but-confusingterminology,“true”and“false”refertoagreementordisagreementwiththegroundtruth(eitherH0orH1),while“positive”and“negative”refertothetestresults(thatis,resultsMandO,respectively).

Theso-calledmaximumlikelihoodestimateofSENisgivenbyTP/(TP+FN),thefractionofeventswithgroundtruthH1(samesource)thatarecorrectlydeclaredasM(match).ThemaximumlikelihoodestimateofFPRiscorrespondinglyFP/(FP+TN),thefractioneventswithgroundtruthH0(differentsource)thataremistakenlydeclaredasM(match).

Sincethefalsepositiveratewilloftenbethemathematicallydeterminingfactorinthemethod’sprobativevalueinaparticularcase(discussionbelow),itisparticularlyimportantthatFPRbewellmeasuredempirically.

Inaddition,testswithverylowsensitivityshouldbeviewedwithsuspicionbecauserarepositivetestresultsmaybematchedoroutweighedbytheoccurrenceoffalsepositiveresults.381

ConfidenceIntervals

Asdiscussedinthemaintext,tobevalid,empiricalmeasurementsofSENandFPRmustbebasedonlargecollectionsofknownandrepresentativesamplesfromeachrelevantpopulation,soastoreflecthowoftenagivenfeatureorcombinationoffeaturesoccurs.(Otherrequirementsforvalidityarealsodiscussedinthemaintext.)

Sinceempiricalmeasurementsarebasedonalimitednumberofsamples,SENandFPRcannotbemeasuredexactly,butonlyestimated.Becauseofthefinitesamplesizes,themaximumlikelihoodestimatesthusdonottellthewholestory.Rather,itisnecessaryandappropriatetoquoteconfidenceboundswithinwhichSEN,andFPR,arehighlylikelytolie. 381Theargumentinfavorofatestthat“thistestsucceedsonlyoccasionally,butinthiscaseitdidsucceed”isthusafallaciousone


128

BecauseoneshouldbeprimarilyconcernedaboutoverestimatingSENorunderestimatingFPR,itisappropriatetouseaone-sidedconfidencebound.Byconvention,aconfidencelevelof95%ismostwidelyused—meaningthatthereisa5%chancethetruevalueexceedsthebound.Upper95%one-sidedconfidenceboundsshouldthusbeusedforassessingtheerrorratesandtheassociatedquantitiesthatcharacterizeforensicfeaturematchingmethods.(Theuseoflowervaluesmayrightlybeviewedwithsuspicionasanattemptatobfuscation.)

Theconfidenceboundforproportionsdependsonthesamplesizeintheempiricalstudy.Whenthesamplesizeissmall,theestimatesmaybefarfromthetruevalue.Forexample,ifanempiricalstudyfoundnofalsepositivesin25individualtests,thereisstillareasonablechancethatthetrueerrorratemightbeashighasroughly1in9.

Fortechnicalreasons,thereisnosingle,universallyagreedmethodforcalculatingtheseconfidenceintervals(aproblemknownasthe“binomialproportionconfidenceinterval”).However,theseveralwidelyusedmethodsgiveverysimilarresults,andshouldallbeconsideredacceptable:theClopper-Pearson/ExactBinomialmethod,theWilsonScoreinterval,theAgresti-Coull(adjustedWald)interval,andtheJeffreysinterval.382Thereareweb-basedcalculatorsforallofthese.383Forexample,ifastudyfindszerofalsepositivesin100tries,thefourmethodsmentionedgive,respectively,thevalues0.030,0.026,0.032,and0.019fortheupper95%confidencebound.Fromascientificstandpoint,anyofthesemightappropriatelybereportedtoajuryinthecontext“thefalsepositiveratemightbeashighas.”(Inthisreport,weusedtheClopper-Pearson/ExactBinomialmethod.)

CalculatingResultsforConclusiveTests

Formanyforensictests,examinersmayreachaconclusion(e.g.,matchornomatch)ordeclarethatthetestisinconclusive.SENandFPRcanthusbecalculatedbasedontheconclusiveexaminationsoronallexaminations.Whilebothratesareofinterest,fromascientificstandpoint,theformerrateshouldbeusedforreportingFPRtoajury.Thisisappropriatebecauseevidenceusedagainstadefendantwilltypicallybebasedonconclusive,ratherthaninconclusive,examinations.Toillustratethepoint,consideranextremecaseinwhichamethodhadbeentested1000timesandfoundtoyield990inconclusiveresults,10falsepositives,andnocorrectresults.Itwouldbemisleadingtoreportthatthefalsepositiveratewas1%(10/1000examinations).Rather,oneshouldreportthat100%oftheconclusiveresultswerefalsepositives(10/10examinations).

BayesianAnalysis

Inthisappendix,wehavefocusedontheSensitivityandFalsePositivesrates(SEN=P(M|H1)andFPR=P(M|H0)).ThequantityofmostinterestinacriminaltrialisP(H1|M),thatis,“theprobabilitythatthesamplesarefromthesamesourcegiventhatamatchhasbeendeclared”.Thisquantityisoftentermedthepositivepredictivevalue(PPV)ofthetest.

382Brown,L.D.,Cai,T.T.,andA.DasGupta.“Intervalestimationforabinomialproportion.”StatisticalScience,Vol.16,No.2(2001):101-33.383Forexample,see:epitools.ausvet.com.au/content.php?page=CIProportion.


129

ThecalculationofPPVdependsontwoquantities:the“Bayesfactor”BF=SEN/FPRandasecondquantitycalledthe“prioroddsratio”(POR).ThislatterquantityisdefinedmathematicallyasPOR=P(H0)/P(H1),whereP(H0)andP(H1)aretheprior(i.e.,beforedoingthetest)probabilitiesofthehypothesesH0andH1.384TheformulaforPPVintermsofBFandPORis:PPV=BF/(BF+POR),aformulathatfollowsfromthestatisticalprincipleknownasBayesTheorem.385

ThisBayesiananalysisoffersamathematicalwaytocombinethetestresultwithindependentinformation—suchas(1)one’spriorprobabilitythattwosamplescamefromthesamesourceand(2)thenumberofsamplessearched.SomeBayesianstatisticianswouldchoosePOR=1inthecaseofamatchtosinglesample(implyingthatitisequallylikelyapriorithatthesamplescamefromthesamesourceasfromdifferentsources)andPOR=100,000foramatchidentifiedbycomparingasampletoadatabasecontaining100,000samples.OtherswouldsetPOR=(1-p)/p,wherepistheaprioriprobabilityofsame-sourceidentityintherelevantpopulation,giventheotherfactsofthecase.

TheBayesianapproachismathematicallyelegant.However,itposeschallengesforuseincourts:(1)differentpeoplemayholdverydifferentbeliefsaboutPORand(2)manyjurorsmaynotunderstandhowbeliefsaboutPORaffectthemathematicalcalculationofPPV.

Somecommentatorsthereforefavorsimplyreportingtheempiricallymeasuredquantities(thesensitivity,thefalsepositiverateofthetest,andtheprobabilityofafalsepositivematchgiventhenumberofsamplessearchedagainst)andallowingajurytoincorporatethemintotheirownintuitiveBayesianjudgments.(“Yes,thetesthasafalsepositiverateofonly1in100,buttwowitnessesplacethedefendant1000milesfromthecrimescene,sothetestresultwasprobablyoneofthose1in100falsepositives.”)

384Thatis,ifpistheaprioriprobabilityofsame-sourceidentityinthepopulationunderexaminationthenPOR=(1-p)/p.385Inthemaintext,thephrase“appropriatelycorrectforthesizeofthepoolthatwassearchedinidentifyingasuspect”referstotheuseofthisformulawithanappropriatevalueforPOR.


130

AppendixB.AdditionalExpertsProvidingInputRichardAlpertAssistantCriminalDistrictAttorneyTarrantCountyCriminalDistrictAttorney’sOffice

RichardCavanaghDirectorSpecialProgramsOfficeNationalInstituteofStandardsandTechnology

WilliamBodziakConsultantBodziakForensics

EleanorCelestePolicyAnalystMedicalandForensicSciencesOfficeofScienceandTechnologyPolicy

MaryA.BushAssociateProfessorDepartmentofRestorativeDentistryUniversityatBuffaloSchoolofDentalMedicine

ChristopheChampodProfessorofLaw,CriminalScienceandPublicAdministrationUniversityofLausanne

PeterBushResearchInstructorDirectoroftheSouthCampusInstrumentCenter

UniversityatBuffaloSchoolofDentalMedicine

SimonA.ColeProfessorofCriminology,LawandSocietySchoolofSocialEcologyUniversityofCaliforniaIrvine

JohnButlerSpecialAssistanttotheDirectorforForensicScience

SpecialProgramsOfficeNationalInstituteofStandardsandTechnology

PatriciaCummingsSpecialFieldsBureauChiefDallasCountyDistrictAttorney’sOffice

AliciaCarriquiryDistinguishedProfessoratIowaStateandDirector,CenterforStatisticsandApplicationsinForensicEvidence

IowaStateUniversity

ChristopherCzyrycaPresidentCollaborativeTestingServices


131

ShariDiamondHowardJ.TrienensProfessorofLawProfessorofPsychologyPritzkerSchoolofLawNorthwesternUniversity

MatthewGametteDirectorofForensicServicesIdahoStatePolice

ItielDrorSeniorCognitiveNeuroscienceResearcherUniversityCollegeLondon

DanielGarnerChiefExecutiveOfficerandPresidentHoustonForensicScienceCenter

MeredithDrosbackAssistantDirectorEducationandPhysicalSciencesOfficeOfScienceandTechnologyPolicy

ConstantineA.GatsonisHenryLedyardGoddardUniversityProfessorofBiostatistics

ChairofBiostatisticsDirectorofCenterforStatisticalSciencesBrownUniversity

KimberlyEdwardsPhysicalScientistForensicExaminerFederalBureauofInvestigationLaboratory

EricGilkersonForensicExaminerFederalBureauofInvestigationLaboratory

ChrisFabricantDirector,StrategicLitigationInnocenceProject

BrandonGirouxPresidentGirouxForensics,L.L.C.PresidentForensicAssurance

KennethFeinbergStevenandMaureenKlinskyVisitingProfessorofPracticeforLeadershipandProgress

HarvardLawSchool

CatherineGrgicakAssistantProfessorAnatomyandNeurobiologyBostonUniversitySchoolofMedicine

JenniferFriedmanForensicScienceCoordinatorLosAngelesCountyPublicDefender

SusanGrossForensicScientistStateofMinnesota


132

AustinHicklinFellowNoblis

RyanLilienChiefScienceOfficerCadreResearchLabs

CindyHomerForensicScientistMainStatePoliceCrimeLab

Anne-MarieMazzaDirectorCommitteeonScience,Technology,andLawTheNationalAcademiesofScience,EngineeringandMedicine

AliceIsenbergDeputyAssistantDirectorFederalBureauofInvestigationLaboratory

WillieE.MayDirectorNationalInstituteofStandardsandTechnology

MattJohnsonSeniorForensicSpecialistOrangeCountySheriff’sDepartment

BrianMcVickerForensicExaminerFederalBureauofInvestigationLaboratory

JonathanKoehlerBeatriceKuhnProfessorofLawPritzkerSchoolofLawNorthwesternUniversity

StephenMercerDirectorLitigationSupportGroupOfficeofthePublicDefenderStateofMaryland

GlennLangenburgForensicScienceSupervisorMinnesotaBureauofCriminalApprehension

MelissaMourgesChiefForensicSciences/ColdCaseUnitNewYorkCountyDistrictAttorney'sOffice

JuliaLeightonGeneralCounselPublicDefenderServiceDistrictofColumbia

PeterNeufeldCo-DirectorandCo-FounderInnocenceProject

AlanI.LeshnerChiefExecutiveOfficerAmericanAssociationfortheAdvancementofScienceandExecutivePublisherofthejournalScience

StevenO’DellDirectorForensicServicesDivisionBaltimorePoliceDepartment


133

LynnOvermannSeniorPolicyAdvisorOfficeofScienceandTechnologyPolicy

DavidSennDirectorCenterforEducationandResearchin

ForensicsandtheSouthwestSymposiumonForensicDentistry

UniversityofTexasHealthScienceCenteratSanAntonio

MatthewRedleCountyandProsecutingAttorneySheridanCountyProsecutor’sOffice

StephenShawTraceExaminerFederalBureauofInvestigationLaboratory

MariaAntoniaRobertsResearchProgramManagerLatentPrintSupportUnitFederalBureauofInvestigationLaboratory

AndySmithSupervisorFirearm/ToolmarkExaminerSanFranciscoPoliceDepartment

WalterF.RoweProfessorofForensicSciencesGeorgeWashingtonUniversity

ErichSmithPhysicalScientistFirearms-ToolmarksUnitFederalBureauofInvestigationLaboratory

NorahRudinPresidentandCEOScientificCollaboration,Innovation&EducationGroup

JillSpriggsLaboratoryDirectorSacramentoCountyDistrictAttorney’sOffice

JeffSalyardsExecutiveDirectorDefenseForensicScienceCenterTheDefenseForensicsandBiometricAgency

HarrySwoffordChief,LatentPrintBranchDefenseForensicsScienceCenterTheDefenseForensicsandBiometricAgency

RodneySchenckDefenseForensicScienceCenterTheDefenseForensicsandBiometricAgency

WilliamThompsonProfessorofCriminology,Law,andSociety

andPsychology&SocialBehaviorLawSchoolofSocialEcologyUniversityofCalifornia,Irvine


134

RobertThompsonProgramManagerForensicDataSystemsLawEnforcementStandardsOfficeNationalInstituteofStandardsand

Technology

VictorWeednChairofForensicSciencesDepartmentofForensicSciencesGeorgeWashingtonUniversity

JeremyTriplettLaboratorySupervisorKentuckyStatePoliceCentralForensicLaboratory

RayWickenheiserDirectorNewYorkStatePoliceLabSystem

RichardVorderBrueggeSeniorphotographictechnologistFederalBureauofInvestigation

XiaoyuAlanZhengMechanicalEngineerNationalInstituteofStandardsandTechnology

President’sCouncilofAdvisorsonScienceandTechnology(PCAST)

www.whitehouse.gov/ostp/pcast

Date post:	06-Mar-2018
Category:	Documents
Upload:	hoangkien
View:	214 times
Download:	1 times