APOPLmarkretrospec0veUsingproofassistantsinprogramminglanguageresearch

StephanieWeirichUniversityofPennsylvania

ThePOPLmarkChallenge

•  Asetofchallengeproblemsmeanttodemonstratetheeffec0venessofproofassistantsinprogramminglanguageresearch

•  IssuedatTPHOLs2005•  BrianAydemir,AaronBohannon,MaJhewFairbairn,J.NathanFoster,BenjaminPierce,PeterSewell,DimitriosVy0nio0s,GeoffreyWashburn,StephanieWeirichandSteveZdancewic

Why?

•  AliJlePLresearchhistory…•  Sinceearly90s,trendinprogramminglanguageresearchtowardssyntac0cmethods

Definethesyntaxandtypesystemofasimplelanguage(STLC+unit)

Nowwhatistheseman0cs?•  Denota0onalseman0cs:mapsprogramstomathema0cal

objects,suchasfunc0ons

•  Opera0onalseman0cs:describeshowprogramsrewritetovalues

Typesoundness

'Well‐typedprogramsdon'tgowrong':

i.e.allevalua0oneitherdivergesorproducesavalidfinalconfigura0on.

Syntac0cSoundnessProof

Alloftheselemmasprovedbysimpletechniques(induc0onorinversion).

Whythistechnique?•  Lowdemandsontheseman0cists

–  Requiresan'opera0onal'viewofprogramexecu0on

–  Easytodefinebecauseitresembleshowthemachineactuallyexecutes

–  RequiresliJlemathema0calmachinery•  "Just"induc0vedatatypes,alpha‐conversion

•  Proofs(ofeasyresults)areeasy–  Seriesofstraighdorwardinduc0ons–  Sameformoflemmaseach0me–  Clevernessisinseengupthetypesystemtherightwaysothattheusualproper0esworkout

Coversmanylanguagefeatures

nontermina0on

higher‐orderfunc0ons

polymorphism

references

con0nua0ons

concurrency

excep0ons

transac0onalmemory

dependenttypes

I/O

objectsaspects

Whatiswrongthismethod?

•  Althoughthemathissimple,therecanbemanycases

•  Syntac0cmethodsmeanintui0oncanfail

•  Easytogetsomethingwronginthedetails,especiallyinthecombina0onoffeatures

To: sml-list@cs.cmu.edu From: Harper and Lillibridge Sent: 08 Jul 91 Subject: Subject: ML with callcc is unsound

The Standard ML of New Jersey implementation of callcc is not type safe, as the following counterexample illustrates:…

The counterexample does contradict a claim by Felleisen and Wright to the effect that the type system is sound; it is my understanding that they have repaired the proof by restricting the language.

13

IngoodcompanyTo: Types List From: Alan Jeffrey Sent: 17 Dec 2001 Subject: Generic Java type inference is unsound

The core of the type checking system was shown to be safe… but the type inference system for generic method calls was not subjected to formal proof. In fact, it is unsound … This problem has been verified by the JSR14 committee, who are working on a revised language specification…

14

Againandagain

From: Xavier Leroy Sent: 30 Jul 2002 To: John Prevost Cc: Caml-list Subject: Re: [Caml-list] Serious typechecking error involving new polymorphism (crash) … Yes, this is a serious bug with polymorphic methods and fields. Expect a 3.06 release as soon as it is fixed. …

15

Ithappenstothebestofus…From: Dimitrios Vytiniotis Subject: very serious bug in one lemma for completeness ... Date: 21 April 2005 To: Stephanie WeirichCc: Simon Peyton Jones As I was typing up the proofs I discovered that the strengthening lemma I have is not correct ... this might affect the whole paper ...Stephanie can we meet if you are around? (otherwise tomorrow ...) :-( ...

-d

Syntac0cmethodscon0nuetobepopular

•  Founda0onforprogramminglanguagestudy

•  Butitcanbetoomuchofagoodthing...

TheStateoftheArtChenandTardi0,ASimpleTypedIntermediateLanguageforObject‐OrientedLanguages,PrinciplesofProgrammingLanguages(POPL),2005

7/20/07 18

TheStateoftheArt

7/20/07 19

PersonalExperiencePublica(on TRlength Heroicgradstudent

JFP07 83pages Dimitrios

ICFP06 59pages Dimitrios

ICFP06 58pages Dimitrios

ICFP05 60pages Geoff,Dan

LICS05 60pages Geoff

TLDI04 51pages Geoff,Dimitrios

WOOD04 49pages Liang

ICFP03 61pages Geoff

Whywrite‐onlyTRs?

Proofsop0mizedforconveyingunderstanding

vs.Proofsop0mizedforconveyingcertainty

i.e.webelievethisistruebecauseweactuallyworkedoutthedetails.Andyoucancheckourdetailsifyouhavethepa0ence

Whohasmorepa0encethanamachine?Exis0ngresearchcommunityoflogicsforexpressingsuchproofs

andtoolsforcheckingthem

Somewerealreadydoingthis…

•  Leroy’sverifiedCcompiler•  Nipkowetal’sformaliza0onofalargepartofJava

•  Appeletal’sFounda0onalProof‐CarryingCodeproject•  Craryetal’smachine‐checkeddevelopmentofatyped

assemblylanguage

•  Harperetal’sformaliza0onofStandardML•  Sewelletal’sformaliza0onofTCP/IP

•  Etc.,etc.

…butnocommonknowledge

•  Whatproofassistanttouse?•  Howtogetstarted?Manuals?Tutorials?

•  Libraries?•  Exis0ngdevelopments?

ThePOPLmarkchallengewasacommunityandinfrastructurebuildingproject

THECHALLENGE,SPECIFICALLY

MetatheoryofSystemF‐sub

Challenge1:Transi0vityofsubtyping

•  Transi0vitymustbeprovensimultaneouslywithnarrowing,whichstates:

•  What'stestedhere:Non‐trivialinduc0veproofs,isola0ngelementsofthecontext

25

If Γ ⊦ S ≤ Q and Γ ⊦ Q ≤ T, then Γ ⊦ S ≤ T.

If Γ, X ≤ Q, Γ' ⊦ S ≤ T and Γ ⊦ P ≤ Q, then Γ, X ≤ P, Γ' ⊦ S ≤ T.

Challenge2:Typesafety

•  ExtendedlanguagewithrecordsandpaJernmatching•  What'stestedhere:Reasoningaboutsyntaxwithvariablenumbersofcomponents–  RecordpaJernsmaybindarbitrarilymanyvariables–  Recordvaluesmaycontainanarbitrarynumberoffields

26

1.  If Γ ⊦ e : T and e → e', then Γ ⊦ e' : T.

2.  If Γ ⊦ e : T, then either e is a value or else e → e' for some e'.

Challenge3:"Anima0on"

•  What'stestedhere:theabilitytoexplorealanguage'sproper0esonpar0cularexamples

•  Solu0onsfor(1)and(2)cancheckaninterpreter•  Solu0onfor(3)isaninterpreter

27

1. Given e and e', decide if e → e'.

2. Given e and e', decide if e →* e' ↛.

3. Given e, find e' such that e → e'.

Evalua0oncriteria

•  Readers:– Adequacyoftheencoding:Isitcorrect?– Obviousnessoftheencoding:Howdifficultisittounderstandadequacy?

•  Writers:–  CluJer,inconvenienceintroducedbythetechnology–  Effortrequiredbeyondapaperproof,evenforexperts

•  Costofentry:– Qualityofdocumenta0on– Maturityoftechnology

28

Whathappenednext?

POPLmarkresults

•  Lotsofinterest!•  15submiJedsolu0onsrecordedonwiki

– 7toolsused(Coq,Isabelle/HOL,Twelf,ATS,Ma0ta,Abella,Alpha‐Prolog)

•  Othersolu0onsdiscussedelsewhere(ACL2,MetaPRL,Nominal‐Isabelle)

"POPLmarktarpit"

•  Techniquesforrepresen0ngvariablebindingcausedthemostheateddiscussion–  7differenttechniquesusedin15solu0ons – Hitapre‐exis0ng,ac0veresearcharea

•  Ourowneffortstounderstandthisissueresultedinnewresearchresults–  EngineeringFormalMetatheory,POPL08Aydemir,Chargueraud,Pierce,Pollack,Weirich

•  Otherpartsofthechallengerela0velyignored– Manydidnotcompletefullchallengewithrecordsoranima0on

Communitydevelopment

•  WeworkedhardtopromotetheuseofproofassistantsamongPLresearchers…

–  Organizedworkshops(4instancesofWMMsofar)–  Developedtutorialmaterial

–  DevelopedalibraryforPLreasoning–  Distributedallofourowndevelopments

–  IntegratedproofassistantuseintoourgraduatePLcourse

Hadtopicksomething…•  DevotedoureffortstoCoqProofAssistant

– Wantedageneralpurposelogic– Wantedamaturepladorm– Construc0velogic,dependenttypeswereaJrac0ve

•  Couldhavechosenotherswithequalsuccess– Exci0ngnewdevelopmentsinthemean0me:Nominal‐Isabelle,Abella,etc.

Itstartedtowork…

•  Morepaperswithmachinecheckedappendicesstartappearing– Somebootstrappedfromourownwork

•  AURA–Zdancewicetal.ICFP2007– Languageforreasoningaboutauthoriza0on– Security‐orienta0onmo0vatesmorecertainty– Sophis0cateddependenttypesystem– MetatheorycompletelydevelopedinCoq– 12.4kLOC

Newtool‐OJ:Sewelletal.Atoolshouldgeneratemanyoutputsgivenasingle“naturallywriJen”defini0onofalanguage

35

Languagedefini0on(ASCII)

Infrastructureformechanizedproofs

(Coq,Isabelle/HOL,Twelf)

Implementa0on(Ocaml,Haskell)

Infrastructureforwri0ngpapers/specs/talks

(LaTeX)

Example:lambdaterms

36

metavar atom, x, y, z ::= {{ coq nat }}{{ coq-equality }}

exp, e, f, g :: '' ::= | x :: :: var | \ x . e :: :: abs (+ bind x in e +) | e1 e2 :: :: app| { e / x } e' :: M :: subst {{ coq subst [[e]] [[x]] [[e']] }}

substitutions single e x :: subst

defn e1 --> e2 :: :: reduce ::'' by

-------------------------- :: ax_app (\x.e1) e2 --> {e2/x}e1

e1 --> e1' -------------- :: ctx_app_fun e1 e --> e1’ e

OJcode

Inductive reduce : exp -> exp -> Prop := (* defn reduce *)

| ax_app : forall (x:atom) (e12 e2:exp), reduce (app (abs x e12) e2) (subst e2 x e12 )

| ctx_app_fun : forall (e1 e_5 e1’:exp), reduce e1 e1' -> reduce (app e1 e_5) (app e1’ e_5).

Example:Typedlambdaterms

37

Definition atom := nat.

Inductive exp : Set := | var : atom -> exp | abs : atom -> typ -> exp -> exp | app : exp -> exp -> exp.

CoqcodeoutputbyOTT

Subs0tu0onoutputLemma eq_atom: forall (x y : atom), {x = y} + {x <> y}.…Fixpoint list_mem (A:Set) (eq:forall a b:A,{a=b}+{a<>b})

(x:A) (l:list A) {struct l} : bool :=…

Fixpoint subst (e_6:exp) (x5:atom) (e__7:exp) {struct e__7} : exp := match e__7 with | (var x) => (if eq_atom x x5 then e_6 else (var x)) | (abs x e5) => abs x (if list_mem eq_atom x5 (cons x nil) then e5 else (subst e_6 x5 e5)) | (app e5 t') => app (subst e_6 x5 e5) (subst e_6 x5 e')end.

38

CoqcodeoutputbyOTT

HowdidthePOPLmarkchallengeimpactmyresearch?

Myresearchmethodshavechanged

•  IuseOTTforallofmytypeseeng–  includingpartsofthistalk–  especiallyexploratory,developmentwork

•  Ifindformalizingthedefini0onsinapaperovenhelpsmyunderstanding

•  Isome0mespopopenaCoqwindowtotryoutsomethoughts

•  Collabora0oniseasierthisway–  Versioncontrol–  Defini0ons,proofstatusalwaysup‐to‐date

•  Newresearchonvariablebinding

Theissuewithvariable‐binding

•  Boundvariablesmustalpha‐vary–  Iden0fy\x.xand\y.y

•  Freevariablesmustbe'sufficientlyfresh'– Capture‐avoidingsubs0tu0one{e'/x}‐‐‐boundvariablesinemustnotbethesameasthefreevariablesine'

– "BarandregtVariableConven0on"

Locallynamelessrep

•  POPL08paperadvocatedtwoideasforvariablebinding

•  Locallynamelessrepresenta0on(oldidea)–  Separateboundandfreevariables– Usenumbersforboundvariables(uniquerepresenta0onofalpha‐equivalentterms)andstringsforfreevariables

•  Cofinitequan0fica0on(newidea)–  Premiseofjudgmentsquan0fiesoverallvariablesexceptforsomefiniteset

–  Stronginduc0onprinciple

POPLmarkchallengeinCoq

43

Locally nameless definitions:

OTT can generate these

Lemmas about free variable and substitution functions

Lemmas for substitution, weakening in judgments

Otherexperiences

•  Rossberg,Russo,Dreyer.F‐ingmodules.TLDI2010

•  13klineCoqdevelopment

•  Usedlocallynamelessapproach

•  400outof550lemmasweretedious"infrastructure"results

LNgen–WorkinProgress

•  BrianAydemirandStephanieWeirich.LNgen:ToolSupportforLocallyNamelessrepresenta>ons.

•  WorkswithOTTtool

•  Generatesandproves'infrastructure'lemmasbasedonlocallynamelessrepresenta0on

•  Examplelemma:iffv(t)=0then[x|‐>u]t=t

Example:STLCdevelopment•  OJ(locallynamelessbackend)–134lines

–  5induc0vedefini0ons(typ,exp,lc,typing,step)–  3func0ons(open,fv,subst)–  1tac0c(tocollectallfreevarsinaproof)

•  Lngen–1533lines–  3func0ons(close,size_typ,size_exp)–  2induc0vedefini0ons(degree,lc_set)–  47lemmas–  2tac0cs,90Hints

•  Handproofs–108lines–  8lemmas(4adeqacy,weakening,subst,preserva0on,progress)

Whatarethose47lemmas

Whyproofgenera0onisok

•  Codegenerators(rightlyso)haveabadname•  Whyisthisareasonablewaytodothings?•  Proof‐irrelevance:don'tcarehowalemmawasproved,onlythatitwasproved

•  lotsofregularstructure–  F‐omega:subs0tutetypesinterms,termsinterms,typesintypes

•  Clearscope:Reasoningrestrictedto5opera0ons–  open,close,subst,fv,lc–  lemmasconcernonlytheseopera0onsandtheirinterac0onswitheachother

Casestudies

•  LNgenprovidedinfrastructurefortwoPOPL2010papers

•  Greenberg,Weirich,Pierce.ContractsMadeManifest– Mostproofsbyhand(60pageTR)–  Trickyreasoningaboutparallelreduc0ondoneinCoq.Replaced8densepagesofTRappendix

•  Jia,Zhao,Sjöberg,Weirich.DependentTypesandProgramEquivalence–  Variedlanguagefor9months,doingproofsbyhand– UsedLNgentocheckresultsinabout2weeks

Contracts

438 terms.v GeneratedbyOTT

3965 infrastructure.v GeneratedbyLNgen

764 prelim.v

3090 thy.v

8257 total

Dependenttypes991 lang.v GeneratedbyOTT 267 langExtra.v 7638 infrastructure.v GeneratedbyLNgen 169 isEq.v 6116 thy.v 2126 thyPP.v 290 progress.v 862 reduc0ons.v 61 isEqSpecifica0on.v 691 isEqBeta.v 2284 isEqC.v 97 inclusions.v

21592 total

ProofsinsteadofTRs(mostly)Venue Mech. How Techreport Heroicstudents

POPL10 some Coq 60pages Michael

POPL10 yes Coq Limin(post‐doc),Jianzhou,Vilhelm

PLPV10 yes Agda Chris

CCS09 yes Coq Aaron,Vilhelm

ICFP08 no Disserta0on Dimitrios

POPL08 yes Coq Arthur,Brian

MFPS07 yes Isabelle/HOL Dimitrios

Wheretofromhere?WhatnextforPLcommunity?

Ac0veresearchintovariablebinding•  JustinCambridge:

– PiJs–NominalSystemT[POPL2010]

– Urban–NominalIsabelle– Kennedy,Benton–StronglytypedCoq

•  Idon'tthinkwehavethecompletestoryyet

Proofengineering

•  Proofengineering– Howtomakesurethatproofsaremaintainable?

– Haven'ttac0caltheoremproversfailedbefore?

•  Idon'tknowtheanswertothisproblem

RoleinEduca0on

•  Pierce:newtextbookusingCoqforgradstudentsatPenn

•  Excellenttoolforteachingaboutproofsbyinduc0on,syntac0capproachtoprogramminglanguagedefini0ons,etc.

•  Whataboutdiscretemath?

Languagedefini0on

•  Whatdoweneedtodotomakesurethatitisstandardprac0cetohaveamachine‐checkedlanguagespecifica0on?

•  Again,heroiceffortsexist…– SML,OCAML(light),Java(light)

•  …butconsensusisnecessary– Languagedesignerswantaccessiblespecs

Goesforlogicstoo…From: Hugo Herbelin Sent: November 2, 2009To: Coq club

> Hi, I have been looking on the web without > success. Is there any paper/tech report > that gives the precise rules of the pCIC as > it is currently implemented in Coq 8.2. > (something like a latex version of Chapter > 4 from the reference manual)

There is a latex version of the reference manual in the Coq source archive and a pdf version at http://coq.inria.fr/distrib/V8.2pl1/files/.

AFAIK there is no other description on paper of the entire set of features of pCIC in its 8.2 implementation. Note however that there is a work in progress by Gyesik Lee and Benjamin Werner on the set-theoretical model of a formulation of pCIC that is very close to Coq.

CONCLUSION

Conclusions

•  Iplantokeeponusingproofassistantsinmydaytodayresearch

TheSuccessofTypedLanguages

•  Itisdifficultforprogrammerstoproveproper0esaboutindividualprograms

•  Instead,languagedesignersproveproper0esaboutlanguagesthatimplyproper0esofallprogramsinthatlanguage

•  Example:Aschemeprogrammermustprovethathisprogramneverexecutes(1+true)

•  AnMLprogrammerknowsthisalready.

Fundamentalidea:Typesafety

•  Milner–Welltypedprogramsdon'tgowrong•  i.e.programsmaintaincertaininvariantsduringtheirexecu0on

•  thoseinvariantsaredescribedbythetypesystem– Func0onscalledwithpar0cularformsofarguments

Howtoprovetypesafety?

•  Sincetheearly90s,typesafetyproved'syntac0cally'

•  Twokeylemmas:– Preserva0on:Ifaprogramtypechecks,andittakesastep,itwills0lltypecheck

– Progress:Ifaprogramtypechecksanditisnotinan(approved)terminalconfigura0onthenitcantakeastep

Currentstateoftheart:OJ

•  Input:Languagedefini0onsinASCII–  Syntax(BNFgrammar)–  Bindingspecifica0ons–  Rela0ons(Typingjudgments,opera0onalseman0cs)

•  Output:mul0pletooldefini0ons–  LaTeX:Typeseengmacros–  Proofassistants:Induc0vedatatypes;func0onsforfreevariablesandsubs0tu0on

•  hJp://www.cl.cam.ac.uk/~pes20/oJ/

65

Whatdidwedo?

•  ComparedsubmiJedsolu0onswithourownexplora0ons:– FJinCoq/Twelf/Isabelle/HOL– ParametricitytheoreminIsabelle/HOL– Damas‐MilnerinNominal‐Isabelle

– Createdourownsolu0onstoPOPLmarkchallengeinCoq