EXT3, Journaling Filesystem The ext3 filesystem is a journaling extension to the standard ext2 filesystem on Linux. Journaling results in massively reduced time spent recovering a filesystem after a crash, and is therefore in high demand in environments where high availability is important, not only to improve recovery times on single machines but also to allow a crashed machine’s filesystem to be recovered on another machine when we have a cluster of nodes with a shared disk. This talk will describe the ext3 filesystem, both its design goals and its implementation. It will explain some of the challenges involved in adding journaling in a way which is completely compatible with existing ext2 filesystems (it is possible to migrate existing ext2 filesystems to ext3 and back again), and will cover the architecture of the implementation, which involves a completely new, generic block device journaling layer in the kernel. 1. Notes 1.1. Original presentation The original presentation of this talk occurred in room A of the Ottawa Linux Symposium, Ottawa Congress Centre, Ottawa, Ontario, Canada on the 20th of July, 2000 at 13:45 local time. This presentation was given by Dr. Stephen Tweedie. 1.2. Presenter bio 1
Transcript
EXT3, Journaling Filesystem
Theext3 filesystemis a journalingextensionto thestandardext2filesystemon Linux. Journalingresultsin massively reducedtimespentrecoveringafilesystemafteracrash,andis thereforein highdemandinenvironmentswherehighavailability is important,notonly to improverecovery timeson singlemachinesbut alsoto allow acrashedmachine’sfilesystemto berecoveredon anothermachinewhenwe haveaclusterof nodeswith a shareddisk.
This talk will describetheext3 filesystem,bothits designgoalsanditsimplementation.It will explainsomeof thechallengesinvolvedinaddingjournalingin awaywhich is completelycompatiblewithexistingext2 filesystems(it is possibleto migrateexistingext2filesystemsto ext3 andbackagain),andwill cover thearchitectureoftheimplementation,which involvesacompletelynew, genericblockdevice journalinglayerin thekernel.
1. Notes
1.1. Original presentationTheoriginal presentationof this talk occurredin roomA of theOttawaLinuxSymposium,OttawaCongressCentre,Ottawa,Ontario,Canadaon the20thof July,2000at 13:45local time.Thispresentationwasgivenby Dr. StephenTweedie.
1.2. Presenter bio
1
EXT3, Journaling Filesystem
Stephenhasbeeninvolvedwith thedevelopmentof theLinux kernelsinceits earlydays.His work hasbeenprimarily on thefilesystemandvirtual memorycode,withmiscellaneouscontributionsall over thekernel.Howeverhenevergoesnearthenetwork code.His recentandcurrentprojectsincludeseveralhigh-endfeaturessuchasraw I/O, fastzero-copy filesystemI/O andhighavailability.
Working for DEC for two years,StephenworkedonVMS kernelinternalsforhigh-availability clusteredfilesystems.He is now employedfull-time by RedHat,which letshim work onLinux exclusively.
Therecordinghasa64kb/sbitrate,32KHzsamplerate,monoaudio(dueto thestyleofsinglemicrophonerecordingused)andhasafile sizeof 35657984bytes.TheMD5sumof this file is: d1aac5c2d7d24123245b3a45956eeb1e
1.4. Creation of this transcript
1.4.1. Request for corrections
This transcriptwasnot createdby aprofessionaltranscriptionist;it wascreatedbysomeonewith technicalskills andaninterestin thepresentedcontent.Theremaybeerrorsfoundwithin this transcript;weaskthatyou reportthemto usingthebugtrackinginterfacedescribedat http://olstrans.sourceforge.net/bugs.php3
1.4.2. Tools used in transcript creation
2
EXT3, Journaling Filesystem
This transcriptionwasmadefrom theMP3recordingof theoriginal presentation,usingXMMS for playbackandlyx (with docbooktemplate)for thetranscription.
1.4.3. Format of transcript files
Thetranscribeddatashouldbeavailablein anumberof formatssoasto providemorereadyaccessto thisdatato a largeraudience.Thetranscriptswill beavailablein at leastHTML, SGML andplainASCII text formats;otherformatsmaybeprovided.
1.4.4. Names of people involved with this transcription
This transcriptionwascreatedby JacobMoormanof theMarbleHorseFreeSoftwareGroup(whosepagesliveat http://www.marblehorse.org)[email protected]
Theprimaryquality assurancefor this documentwasperformedby StephanieDonovan.Shemaybereachedat [email protected]
1.4.5. Notes related to the use of this document
Thisdocumentis distributedin thehopethatit will beuseful,but WITHOUT ANYWARRANTY; without eventheimplied warrantyof MERCHANTABILITY orFITNESSFORA PARTICULAR PURPOSE.While quality assurancecheckson thistranscriptwereperformed,it wasnotcreatednorcheckedby aprofessionaltranscriptionist;thetechnicalaccuracy of this transcriptis neitherguaranteednorconfirmed.Pleasereferto theoriginal audiorecordingof this talk in theeventconfirmationof thespeaker’s actualstatementsareneeded.
1.4.6. Owner ship of the content within this transcript
Thesetranscriptslikely containcontentowned,undercopyright, by theoriginalpresentationspeaker; pleasecontactthemfor licensingrequests,but dosoin apolite
3
EXT3, Journaling Filesystem
manner, please.It mayalsobeusefulto contactthecoordinatorfor theOttawaLinuxSymposium,theoriginal venuefor this presentation.All trademarksarepropertyoftheir respectiveowners.
1.5. Markup used in this transcript
1.5.1. Time marker s
At theendof eachparagraphwithin thebodyof this transcript,a timeoffsetis listed,correspondingto thatpoint in theMP3recordingof thepresentation.This timemarkeris emphasized(in documentformatsin whichemphasisis supported)andis placedwithin bracketsat theveryendof eachparagraph.For example,[05m, 30s] statesthatthis paragraphendsat thefive-minute,thirty-secondmarkin theMP3 recording.
1.5.2. Questions and comments from the audience
Theserecordingswerecreatedusingabud microphoneattachedto thespeakerduringtheir presentation.Dueto theinherentrangelimitationsof this typeof microphone,someof thecommentsandquestionsfrom theaudienceareunintelligible.In caseswherethespeaker repeatstheaudiencequestion,thequestionshallbeomittedandamarkerwill beleft in its place.Eventswhichhappenin theaudienceshallbebracketed,suchas:[Theaudienceapplauds.]
Further, in caseswheretheaudiencecommentsor questionsarenot repeatedby thespeaker, they shallbeincludedwithin this transcriptandshallbeenclosedwithindoublequotesto delineatethatthestatementscomefrom theaudience,not from thespeaker.
1.5.3. Editorial notes
4
EXT3, Journaling Filesystem
Theeditorof this transcript,thetranscriptionist(if youwill), andthequality assuranceresourcewhohaveexaminedthis transcriptmayeachincludeeditorialnoteswithin thistranscript.Theseshallbeplacedwithin bracketsandshallbegin with ’ED:’. Forexample:[ED: Theauthoris referringto slicedcheese,notgratedcheese.]
1.5.4. Paragraph breaks
Theparagraphbreakswithin this transcriptareverymucharbitrary;in many casestheyrepresentpausesor breaksin thespeechof thespeaker. In othercases,they havebeeninsertedto allow for enhancedclarity in thereadingof this transcript.
1.5.5. Speech corrections by the speaker
During thecourseof thetalk, thespeakermaycorrecthimselfor herself.In thesecases,thecorrectedspeechwill beplacedin parenthesis.Thereaderof this transcriptmayusuallyignoretheparenthisedsectionsasthey representcorrectedspeech.Forexample:My auntoncehad(a dognamedSpot,sorry)acatnamedCleopatra.
1.5.6. Unintelligib le speec h
In sectionswherethespeechof theauthoror audiencehasbeendeemeduseful,butunintelligibleby thetranscriptionistor by thequality assuranceresource,amarkerwillbeinsertedin theirplaces,[unintelligible]. Severalattemptswill bemadeto correctwordsandphrasesof this nature.In caseswheretheunintelligiblewordsor phrasesareclearlynotof importanceto themeaningandunderstandingof thesentence,they maybeomittedwithoutmarker insertion.
2. Transcript
5
EXT3, Journaling Filesystem
Wehadanoverview this morningaboutall thedifferentfilesystemsin Linux androughlywhatdifferentoptionsthereareaboutthesedays.I’m goingto just betalkingpurelyabouttheEXT3 filesystemin general.I’ ll alsotalk a little bit aboutsomeof thespecificthingsthatwe’vebeenworkingon in thevirtual memorysystemthatparticularlyaffect journalingfilesystemsandthework thatwe’vebeendoing.Forexample,thingslikeallowing filesystemsto reservememoryin clean,deadlock-safeways.But really thebulk of thetalk will beaboutthis EXT3 filesystemandalsoaboutsomeof the(reallyaboutthe)implementationdetails;how theinternalsof thefilesystemabstractout thejournalingfrom theactualfilesystemoperations.[00m, 54s]
And why havewedoneall of this EXT3 stuff anyway?Well, really thereareseveralmotivations.Firstof all, peoplestill likeEXT2; they still trustEXT2. EXT2 is a fairlywell provenfilesystem.It doesn’t haveall thebellsandwhistlesof someof thenewerfilesystems.It doesn’t have thesmallfile efficiency of reiserfs.It doesn’t have thedirectoryscalabilityof XFS,but it is aprovenworkhorsefor Linux. And mostimportantly, therearea ratherlot of usersout therewho havegot existingEXT2filesystems.And moreeveryday. [01m, 40s]
And someof theseEXT2 filesystemsaregettingreally ratherbig. Even24 monthsago,therewerepeoplebuilding 500gigabyteEXT2 filesystems.They takea long time tofsck. I mean,really. Thesearefilesystemsthatcantake threeor four hoursjust to mkfs.Doingaconsistency checkon themis aseriousdown time.Sotherealobjective inEXT3 wasthis simplething:availability. Whensomethinggoesdown in EXT3, wedon’t wantto have to go througha fsck.Wewantto beableto rebootthemachineinstantlyandhaveeverythingniceandconsistent.[02m, 23s]
And that’sall it does.It’ saminimalextensionto theexistingEXT2 filesystemto addjournaling.And it’ s really important,EXT2 is theworkhorsefilesystem.It’s thestandardstablefilesystem.Wedon’t wantto turnEXT2 into anexperimentalfilesystem.For onething,usersexpectto haveEXT2 thereasademonstrationof howto codefilesystemsfor Linux. It’ s asmall,easilyunderstoodfilesystemwhichdemonstrateshow to do all of thetalking to thepagecache,which haschangedin 2.4,all of thelocking in thedirectoryhandling,which haschangedin 2.4.All of thesechangesin theVFS interfaceandtheVM interfacethatfilesystemshave to dealwithareshowcasedin EXT2. Sotherearemultiple reasonswhy wereally donot wanttostartmakingEXT2 into anexperimentalfilesystem,addingall sortsof new
6
EXT3, Journaling Filesystem
destabilizingfeatures.And sotherealgoalfor EXT3 wasto provide theminimalchangesnecessaryto providea completejournalingsolution.[03m, 26s]
Soit providesscalingof thediskfilesystemsizeandit allowsyou to make largerandlargerfilesystemswithout thefsck penalty, but it’ snotdesignedto addthingslikevery,very fast,scalabledirectories(to Linux) to EXT2. It’ snotdesignedto provideextremelyefficient supportfor extremelylargefilesandextent-mappedfilesandthistypeof thing thatpeoplehavebeentalkingaboutfor EXT2. But really thegoalis toprovide thatonepieceof functionalityalone.[03m, 55s]
And oneof themostimportantgoalsfor thewholeprojectwasto provideabsolute,total,completebackwardsandforwardscompatibilitybetweenEXT2 andEXT3. YoucantakeanexistingEXT2 filesystem,throw a journalfile ontoit, andmountit asEXT3. There,youhavea journalledfilesystem.This laptop,I installedRedHat 6.2onit, it formatsall of thesepartitionsasEXT2. I’veaddedacoupleof journalfilesandnow it’ sall runningEXT3. [04m, 28s]
Betteryet, if youunmountanEXT3 filesystem,thenit marksthatfilesystemashavingbeencleanlyunmounted.Thejournaldoesn’t needto berecoveredafteracleanremount,soall theseextradatastructuresyou getfor journaling,for dealingwithreboots,arenot relevantif thefilesystemhasbeenunmountedcleanly. If you doacleanunmountof anEXT3 filesystem,youcanthenmountit againasEXT2 andEXT2 justdoesn’t care.It’ s completelycompatiblein bothdirections.[05m, 02s]
If youhaveanEXT3 filesystemthataftera crash,for example,andthere’s a journalontherewhich is activeandneedsto bereplayedontothefilesystem,thenin thatparticularcaseyouneedto have recovery. Youcannotmountit asEXT2, becauseto do so,youwouldgetaninconsistentfilesystem.Youwouldcorruptall of theinformationthat’s inthejournal.SoEXT2 hasfor sometimehadasetof compatibilitybits in thesuperblockwhich let you say, for example:whatfeaturesarein thefilesystem?Sothatversionsofthekernelwhichdon’t understandaparticularfeaturewill not try to mountafilesystemthatit can’t understand.Soif you’vegotanactiveEXT3 filesystem,theflagsetin thatsuperblockwhich says:don’t you daremountthisasEXT2; it’ snot compatible.AndEXT2 will cleanly, will bequitecareful;I don’t understandthatbit, I’ ll refuseto mountit. But assoonasyou’veunmountedit, thatbit getscleared.[05m, 56s]
Question?
7
EXT3, Journaling Filesystem
“You answeredit.”
Okay. And similarly thefsprogs,e2fsckcanusethosebits to work outexactlywhetheror not this perfectlynormal-lookingEXT2 filesystemactuallyhasanEXT3 journalandwhetheror not thatjournalneedsrecovery. [06m, 14s]
Sowehavecompletecompatibility, bothin termsof theon-diskformat,sothatanunmountedEXT3 filesystemdoeslook exactly likeEXT2, but alsoin termsof thefunctionality. Soall of theexisting thingsthatwehave in EXT2... thingslike thepersistentinodeattributes,youcansetonaninode.For examplein EXT2, youcansetadirectoryinodeto besynchronous,sothatall of theupdateson thatfilesystemhavesynchronousmetadataupdates.[06m, 41s]
And youcansetthatattributeon amail spoolto getDSDsynchronousupdatecompatiblebehavior for your mail files.Thingslikesendmailcanmakeassumptionsaboutconsistency aftera reboot.And if youneedthatkind of consistency, you candothatin EXT2. All thoseattributesareall exactly therein EXT3, becausetheEXT3sourcecodestartedoff by metakingacopy of theentireEXT2 directoryandcopying itinto thedirectorycalledext3 andthendoingaglobalsearch-and-replacefor alloccurrencesof ext2 andreplacingthatstringwith ext3. [07m, 13s]
It’ sexactly thesamesourcebaseit startedoff from. Theonly reasonit wasmadeinto aseparatesourcetreewassothatI couldhave testboxesthatrunEXT3 developmentcodeon my testpartitionwithouthaving to run thatsamedevelopmentcodeon my rootfilesystem,which kind of mademea little bit nervous.I didn’t wantto do that.[07m,31s]
SowehaveEXT3 asaseparatefilesystemsimply for thatreason,to isolatethenewcodefrom theold, stablecode.But apartfrom that,I meanit’ sexactly thesamesourcebasethat’sbeenusedfor thetwo. There’s no lossof existing functionality. And inparticular, theguaranteeaboutjournalingconsistency coversall of theexistingfunctionality. [07m, 51s]
Soeventhingslikequotasareguaranteedto beconsistentaftera rebootwithjournaling.Soif youupdatea file, write to or extendafile, or truncateafile, thequotaoperationsthatgoon alongsidethatoperationareguaranteedto beconsistentwith thecontentsof thequotafile. You neverhave to run aquotacheckaftera rebooton EXT3,
8
EXT3, Journaling Filesystem
just asyouneverhave to run a fsck.[08m, 16s]
Thereareanothercoupleof subtleissuesaroundrecoverywhicharereally notat allhandledin any waywhatsoeverby existingfilesystems.Themainoneof theseisorphanedfiles.Orphanedfiles is this conceptthattheGFSpeopleweretalking aboutearlierasbeinganastycase.Thatyoucanhavefileswhicharedeletedfrom thefilesystem,but which arestill openby processes.And thesemanticsarethatwhenthatprocessdies,you wantto closeandyouwantto deletethefile, remove it from disk,returnall thespaceto thefilesystem.[08m, 53s]
Well, that’sall well andgood.Thetroubleis that,thatfilesystemstatethathasthefile(closed,sorry)deletedbut still open– that’sacompletelyconsistentstatefor thefilesystem.Soin thefirst versionsof EXT3, if youhaveafilesystemin thatstate,it’ sperfectlyconsistent.Theunlink of thatfile is asingletransactionon thedisk.Thattransactiondid not reclaimthediskspacebecauseit didn’t needto, thefile’s still open.Sowe’vegot this completelyconsistenton-diskstatein which thereis afile whichexistson thediskbut isn’t in thedirectorystructureanywhere.[09m, 27s]
Now obviouslyaftera reboot,youcanbeprettysurethattherebootalsokills aprocessthathadthatfile openandsoweneedto preserve thesemanticsthatkilling theprocessdeletesthefile. Sowehave to havesomewayof dealingwith theseorphanedfilesandthat’snew functionalitythatwe just don’t have to dealwith in EXT2, becauseEXT2assumesthatthere’salwaysafilesystemcheckthereto cleanup thesethingsafterareboot.Sotherearea few thingslike thatin EXT3 thatweneedto dealwith afterarebootto makesurethatconsistency is complete.And so,really, this is thegoal:absoluteconsistency of thefilesystemin every respectaftera reboot,with no lossofexisting functionality. [10m, 08s]
Sohow’s it actuallyimplementedinternally?Whatdoesthissourcelook like?Well youseethefirst thing wedid is just take theEXT2 filesystemandturn it into EXT3. That’snot theonly new filesystemsubdirectorythattheEXT3 patchesprovide.EXT3 alsoprovidesanew (subdirectoryand)filesystemdirectoryundertheLinux sourcetreecalledJFS.[10m, 35s]
That’s thejournalinglayerfor EXT3; andthat’sentirelyindependentof theEXT3filesystemitself. It’ s acompletelyabstractjournalinglayerwhichallowsyou to makearbitraryblockdevicemodificationsin thebuffer cacheandhave thoseobey
9
EXT3, Journaling Filesystem
transactionalsemantics.Sothatyou canmakearbitrarytransactionson arbitraryblockdevicesandfilesystemtransactionsarejustoneexampleof thingsyoucando with that.[10m, 59s]
It wasexplicitly writtensothat,for example,if you hada logical volumemanagerwhichwantsto makeanumberof changesoveranumberof differentblockdevicesandmake thosechangescompletelyatomic,with respectto reboot.Sothat,for example,you areaddingawholepile of logical volumesor you’redeletinga pile of logicalvolumesandyou’reupdatingit acrossmultipleblockdevicesfor astripeset.[11m, 21s]
TheJFSlayerthatweaddedfor this is perfectlycapableof beingusedfor somethinglike thataswell; it’ snot restrictedto theEXT3 filesystem.And in particulartheEXT3filesystemdoesnotknow anythingaboutjournaling.Journalingis separate.EXT3doesn’t have thejournaling;all it knows is transactions.It says:hereis thebeginningofasetof blockdevicemodifications;I’m goingto modify this block devicewhichcontainsmy filesystemandI’m goingto tell you thatthesefiveblockdeviceupdatesform asingletransaction.And it tells thatto thejournalinglayerandthejournalinglayeris responsiblefor makingsurethosefiveupdateseitherall appearin theblockdeviceaftera reboot,or noneof themappear. And thejournalingis doneinsidethat.EXT3 doesnotknow aboutjournaling.[12m, 13s]
Theonly placeswhereEXT3 hasto interactwith thejournalinglayerareto tell itwherethesetransactionsstartandstopandwhichupdatesbelongto which transaction.And also,to managethediskspacewhich is usedby theinodewhichcontainsthejournal.And thejournal in theJFSlayercanbeon any inodein any filesystemor it canbeon anarbitrarysub-range,setof contiguousblocksonany blockdevice. It doesn’tevenhave to beon thesamedevice thatyourfilesystemis on.And youcanhavemultiplefilesystemssharingthesamejournal,if you want.TheJFSlayerwill copewith all of that.[12m, 49s]
Theonly otherthing...sotheonly thingswehave to do is managethetransactionsandmanagethejournalinode.And oneof thethingsinvolvedin that,is thatthefilesystemis responsiblefor askingthejournallayerto do journalrecoveryafteranuncleanreboot.And asI saidearlier, onceyou’vedoneall that,onceyou’veunmountedthething, theEXT3 filesystemtells thejournalingto closedown, cleanup thejournalandmarkeverythingasconsistent.And onceit’ sdonethatandthejournalinglayerhas
10
EXT3, Journaling Filesystem
said:find all of yourupdatesareconsistent,andthis, thejournalhasbeenemptiedout,Thefilesystemcanthenjust setaflag in thefilesystemthatsays:okay, weno longerneedto recoveranything;don’t worry aboutthejournalandjust feel freeto mountit asEXT2. Sothat’sbasicallywhatwehave in thedesign.It’ s two completelydifferentsetsof code;onethat’s theabstractjournalinglayerandone,asimplesetof modificationsto EXT3 to addtransactions.[13m, 48s]
Soif we look at this layerthat’saddingjournaling,whatdoesit look like?Whatdoesitprovide?Well whatit doesis, it exportsanAPI whichallowsyouaddtransactionsontoany blockdevice.SojustasEXT3 doesn’t understandthefirst thing aboutjournaling,itdoesn’t needto, theJFSlayerdoesn’t understandthefirst thing aboutfilesystems,itdoesn’t needto. All of thefilesystempropertiesaredealtwith in EXT3. [14m, 16s]
All of thejournalingis donewithin theJFSlayer. It providesanabstractconceptof alog; it allowsyou to registera journalwith theJFSlayer. And whenyou registerthatjournal,you’vegot two choices,youcaneithersay:here’s someinodeonsomefilesystem;pleaseusethecontentsof this inodeasa journal.Thatmustbea blockdevicefilesystembecausetheJFSlayerassumesthatit canalwaysdo amappingbetweenanarbitraryblock insidethatinodeandtheblockon disk.[14m, 50s]
Soit says:createoneof thesejournals,eitherin aninodeon ablockdevice. It doesn’tcarewhichyou useandyou don’t evenhave to have thejournalon thesamedevice thatyou’regoingto bedoingtheupdateson.Soyoucanhaveafilesystemoverhereandjournalit to aseparatespooldiskover there,if youwantto. TheJFSlayerwill bequitehappy with that.In fact,youcanevenhavemorethanoneblockdevice journalingtothesamedisk, if you want.TheJFSlayeris quitehappy aboutthat.[15m, 20s]
It provideswrite orderingguarantees.All theway throughtheI/O layersin thekernel.Soit makesall theseguaranteesthat,if you havea transactionwhich is in progress,buthasnotyet beencommitted,theJFSlayerprovidesaguaranteethatnotoneupdateofthatparticulartransactionwill hit themaindiskuntil you’vedonethecommit.Itprovidesall thatguarantee,but it doesn’t necessarilytell you whentheupdateswill hitthedisk.TheJFS’updatesarestill write-behind.[15m, 57s]
Soin otherwords,you’renotdoingtransactionssynchronously;you’renotsaying:wellI’ ll doall of theseupdatesandthatresultsin thecreationof anew directoryon thedisk.And thenall of theupdatesI’vemadeto thediskarecommittedinto thejournalandI
11
EXT3, Journaling Filesystem
return.[16m, 12s]
TheJFSlayermaintains,in cache,a list of all of theupdateswhich form any particulartransactionandit will do, in its own sweettime, thenormalwrite-behindthatthebufferlayeris alreadydoingon this. It’ ll just write behindin somefuturetime; we’ll makesureall of thosefutureupdateshit thedisk.But it will makeguaranteesabouttheordering,sothatwhenthey do hit disk, thetransactionis eitherall there,or not thereatall. [16m, 37s]
Now journalingin many casesis verysimilar to databases.[16m, 45s]
[Thespeakercallsonanaudiencememberfor aquestion.]
“Doesthejournalinglayerprovideanorderingguaranteebetweentransactions?Cantransaction47 be...whenyou recover, cantransaction47 exist in therecoveredstatewhentransaction46 doesnot?” [17m, 08s]
Thequestionis: is thereanorderingguaranteesin theJFSlayer?Yesandno.This isactuallya reasonablycomplicateddesignissue.And it’ ssomethingwhich in particularmakesahugeamountof differencewhetheryou’re runningona singlenodefilesystemor ashareddisk. It’ soneof thereasonsthatGFSimplementsthingsverydifferently,internally, to EXT3. TheJFSlayerin EXT3... its API doesnotmakeany guaranteesabouttransactionordering.If youhavea transactionwhichupdatesblocksone,two andthreeon thediskandanotheronewhichupdatesblocksfive,six andseven,theAPIdoesn’t giveyouany orderingguaranteebetweenthose.[18m, 00s]
Now on somethinglikeGFS,if it’ sdoingjournaling,that’s really importantthatthere’sno orderingguaranteebecausein GFS,you’vegot to beableto releaseadiskblockbackto diskasquickly aspossiblein orderto relinquisha lock, which is requiredbysomeothernode.On this local diskfilesystem,youdon’t have that.Soon a localdiskfilesystem,it’ squitelegitimateto batchall theseupdatesoff into very largetransactionsandjust sendthemall outat once.Thatworksreally efficiently. [18:36]
Theonly placeit breaksdown on a local disk is if you’vegotsynchronousupdates.Ifyou do, for example,anfsync()on afile, or if you opena file asO_SYNC,andin thatcaseif you haveabsolutewrite orderingguarantees,thenin orderto flushthis onelittlefile out to disk,sayit’ sa mail spool,andyou’redoinga fsync()for somefile that’s justarrivedoff thenetwork; in orderto syncthatto disk, if you’vegotwrite ordering
12
EXT3, Journaling Filesystem
guarantees,thenyou’vegot to syncall previoustransactionsandcommitthemto diskaswell. And that’sexpensive,becausethathurtsthelatency of thosesyncs.It doesnotchangeyour bandwidth;it’ sactuallymoreefficient in termsof diskbandwidthandthroughputto batchyour transactionsoff in largechunks.[19m, 18s]
SotheAPI thatJFSexportsdoesnot makeany guaranteesaboutwrite orderingconsistency. Internally, it batches(all of thetransactions)all of theupdateswhich aremadeby thefilesystem...it just batchesthemupsequentiallyinto big, compoundtransactionsandputsthemout to diskasasingleunit. Sotheimplementationmakes...doesactuallyhappento preservewrite orderingin all cases.[19m, 45s]
Thatis notguaranteedby theAPI, andoneof thethingsthatweneedto do oncethecoreis out thereandbeingused...onceeverythingis upandrunning,includingall oftheperformancestuff that’sstill a work in progress,is to doprofiling to find outwhetheror not thereareapplicationswhich reallydo needto havefine-grainedcommittingof transactions,sothatwhenyou fsync(),you just...youknow you’vegottransactionsthree,four, five,six andseven,andtransactionsix suddenlybecomesrelatedto anfsync(),soyouhave to committhatone.Doesit actuallymakeadifferenceto performanceto beableto committransactionsix without having tocommittransactionsthreeto five?If thatturnsout to bethecase,thenI will beabletodo thatin thefuturewithoutchangingtheAPI. TheAPI doesn’t make thatguarantee;it’ s just anoptimizationinternally. [20m, 36s]
But apartfrom thatwhole,theright orderingguaranteesaremadeandhow it doesthisis fairly simple.Journalingfilesystemsare,in mostcases,verycloselyrelatedtodatabasejournaling.But with somevery, veryspecialcases.In databases,typically alocking databaseis characterizedby thestepsit hasto go throughto recover thestateofthedatabaseafteracrash.And it’ scharacterizedby whetheryouhave to do undo’sorredo’s.[21m, 19s]
Soin thecaseof anundologging,thatmeansthatwhatyoudo is thatyou put into thejournalall of theold stateof thebuffersthatyou’remodifyingandthenyoucanwritethenew stateto thedisk.And if youcrashbeforethetransactionis committed,thenyoucanundothemodificationsyoumadeondiskby copying theold contentsfrom thelog.Sothatthelog containstheinformationnecessaryto undoincompletetransactions.[21m, 46s]
13
EXT3, Journaling Filesystem
Or you candoredologging,which is thatthenew datais written into thelog andyouleave theoriginaldataon thedisk, in its mainlocation.And thenafteracrash,anytransactionswhichareincomplete,theoriginal copy is still on disk,soyou don’t haveto doanything.And it’ sonly completetransactionswhich exist in thelog andyou haveto replaythoseinto themainfilesystem.[22m, 08s]
In almostall cases,journalingfilesystemsjustusesimplecaseof doingredologging.Sobasicallyeverysinglemodificationthat’smadeto thefilesystemwill bewritten tothelog first. And only onceit’ scommittedto thelog, not just written in thelog, butcommittedto thelog, areweallowedto updatethemaincopy on disk.And that’s (whatEXT2 does)whatEXT3 does,soall of thosewrite orderingguaranteesareprovidedbytheJFSlayer. [22m, 36s]
SothatJFSlayercontrolsthesevarious,differentthings.It controlstransactioncommitandthecommitof a transactioninvolveswriting all of thethingswhich thattransactionmodifiedto thejournal,andthenwriting acommitrecord.It’ snot sufficient just towrite thething to thejournal,becausethere’sgot to besomemarkin thejournalwhichsays:well, (hasthis journalrecordactually)doesthis journalrecordactuallyrepresentacompleteconsistency to thedisk?And thewayyoudo thatis by having someatomicoperationwhich marksthattransactionasbeingcompleteondisk.[23m, 14s]
Now, disksthesedaysactuallymake theseguarantees.If youstartawrite operationto adisk, thenevenif thepower fails in themiddleof thatsectorwrite, thediskhasenoughpoweravailable,andit canactuallystealpower from therotationalenergy of thespindle;it hasenoughpower to completethewrite of thesectorthat’sbeingwrittenright now. In all cases,thedisksmake thatguarantee.[23m, 41s]
Sothefundamentalthing abouttransactionsare:at theendof writing thenew contentsof thetransactionto thelog, wewrite asingle512-bytesectorto thedisk,whichcontainswhatevermagicnumbersto identify asit asaparticulartypeof block; sosaythis is a commitblockandit will containa sequencenumberthatmatchesall of thetransactionsthathavegonepreviously, sothatit doesn’t getconfusedbetweenthat,whatyou’vebeenwriting there,andtheold contentsof thelog, previously. [24m, 14s]
Thatsinglewrite of thatonesectoron diskmarkstheentiretransactionasbeingcomplete.And soall of thatwrite ordering,thewrite orderingwithin thejournal,thatsaysthatthejournalhasto bewrittenandcompleteondiskbeforeyouhaveto write the
14
EXT3, Journaling Filesystem
commitrecord...that’sall handledby JFS.Thewrite orderingthatsays:youhave towrite thecommitrecordbeforeyou write any of theblocksbackto themainfilesystem.That’sall handledby theJFS.[24m, 38s]
Obviously there’sonly a limited amountof spacein thatlog. Sotheold transactioncheckpointing...Checkpointingis theprocessof flushingall thecontentsof thelog outto themaindisk.That’shandledby theJFSlayerandthat’sactuallyreally important,becauseif you think aboutit, thetransactionis committedon diskassoonaswe’vewritten thatcommitrecordin thelog. But oncethatcommitrecordhasbeenwritten tothelog, theonly copy of thedatathatthetransactionhasjustwritten is in thelog; themaincopy ondiskstill hastheold version,sowecannotthrow awaythatdatain theloguntil wehavewritten it, copiedit back,ontodisk.And that’scalledcheckpointing.[25m, 19s]
That’swhatallowsusto re-usebitsof thelog. We take thecontentsof thelog, makesurethecopieson diskareall up-to-date,andat thatpoint,wecantrim thetail of thelog. All of thatis handledby theJFSlayer. All of thewrite-behindis handledby theJFSlayer. It hasits own setof timeoutsandlinks itself into thebuffer cachewrite-behindlayersandsoall of thatis handledcompletelytransparentlyto thefilesystem.And also,theJFSlayer, for performancereasons,triesto makeeverythinggo asasynchronouslyaspossible.It neverstallsthings;it never triesto do morecopiesthanit needsto. [25m, 55s]
So,for example,whenwe’re doingjournaling,thefilesystemmaysay:takesomebuffer that’s in memoryandI’m goingto write this to disk for thatapplicationthereandjournalit. And thejournalingcodehasto makesurethatblock getswritten to thejournalfirst, andthen,afterthecommit,it goesto its mainlocationon disk.And thejournalinglayerwill dozero-copy from that;it will actuallycreateanew I/O requestthatpointsto theold diskbuffer locationandusethatto journalthedatato thejournalfile, without copying thedatablock.Now all thatkind of thing is handledby theJFSlayer. [26m, 33s]
[Thespeakercallsonanaudiencememberfor aquestion.]
Thequestionis: doesit makesenseunderextremememorypressureto throw awaydatathat’sbeenwritten to thejournal,but hasn’t yet beenwritten to disk.Absolutelynot,becauseif it’ swritten into thejournalandyouwantto throw it away, it’ smuch,much
15
EXT3, Journaling Filesystem
moreefficient to just write it to disk thandoinganythingelse.Actually on thenextslides,I’ ll talk aboutreservations,whicharetheway thatyou limit theamountofoutstandingdatathat’sbeingusedby thetransactionlayeratany point in time.And thatreservationsystemis sufficient to reducethememorypressure.[27m, 21s]
[Thespeakercallsonanaudiencememberfor aquestion.]
Sothetwo questions:first of all, thesectorsizethatthedisksguaranteeto beatomicissmallerthan(thesectorsize)theblocksizethefilesystemuses.Yes,but for thecommitblock, I only evercareaboutthefirst 512bytesof thatblock. If thecommitblock’sfirst512bytesareup-to-dateondisk, thenthat’sassumedto mean(thesector, sorry)theentiretransactionis committed.And youhave to bevery, verycarefulnot to havecritical commitinformationthatspansa512-bytesectorboundary. As for writeordering,absolutely. Weneedto makesurethatevenwhenwesubmitmultipleasyncI/O requeststo thedisk, thediskdoesn’t allow usthewrite to reorderthingsin suchawayasthatthecommitblock hits thediskbeforetheothertransactionblocks.And Iwill comebackto thatpoint,becauseit’ sactuallya reallynastypoint for performance.(I’ ll comebackto that,yes.)[28m, 36s]
Oneotherthing thatI’ ll sayaboutthis in termsof asynchronousbehavior is thatthejournalinglayeris really careful...(Ah, comeback.)Thejournalinglayeris reallycarefulto makesurethatthingsdon’t stall unnecessarily. Sothatmeansthatwhenwestartcommittinga transaction,(wedon’t stop)wedon’t stall thefilesystemitself. Andcommittinga transactionmeansthatwe take,okay, I’m goingto takea snapshotof theentirefilesystemstateat this point in timeandI’ ll startcommittingthatstateto disk.But thefilesystemis still allowedto makenew copiesof thedata.Thefilesystemis stillallowedto modify thevirtual blockdevice in thebuffer cache.While it doesthat,wehave to keeptheold contentsof thatsnapshotpresentin memory, sothatwecancommitit to disk.[29m, 41s]
And soin thatparticularcase,theJFSlayerprovidesacopy-on-writemechanismsothatif a new filesystemrequestcomesalong,thatwantsto modify ablock thatwearein theprocessof committingbut haven’t finishedcommitting,thenwemakeacopy ofthatbeforethefilesystemis allowedto modify thebuffer. And thatmeansthatthediskI/O for committinga previoustransactioncango on in parallelwith thefilesystemoperationsfor thenew transaction.Thatis oneof thethingswedo in theJFSlayerto
16
EXT3, Journaling Filesystem
makesurethattheconcurrency of thesystemis ashighaspossible.Therearenosynchronoustransactionsin theJFSlayer, at all. Theonly wayyoucangetsomethingsynchronousis if yousay:well I actuallywanttheapplicationto wait until this thing ison diskbecausetheapplicationhasdonesomethinglikeanfsync().[30m, 28s]
SoJFSprovidesall this functionalityandit providesit to theuser, wherein this casetheuseris somethinglike theLVM layeror in this case,theEXT3 layer. And it hasaniceabstractAPI for exportingthis functionality. Everythingis expressedin terms,notof transactions,but of handlesandto make this distinctionclear, ahandlerepresentsonesingleoperationthatmarksaconsistentsetof updateson thedisk.[30m, 59s]
Soa handlemightbesomethinglikeacreateandthecreatehasto go throughadirectory, addadirectoryentryto thatdirectory, modify thetimestampon thatdirectory, modify thesizeof thedirectory;it hasto allocateanew inode,andit thenhasto modify theinodetablefor that;it hasto modify thesuperblockto changethenumberof inodesin thatgroupin thesuperblock;andit hasto marktheinodebitmapasbeingchanged.And all of theseoperationsfor asingle,consistentoperationupdatein thefilesystemaredonewith a singlehandle.[31m, 37s]
But ahandleis notnecessarilythesamething asthejournalingtransactionon disk,becausethejournalinglayerwill allow multipleupdateslike this to bebatchedinto asingletransaction.So,to make this distinctionquiteclear, thetransactionon thedisk isnotnecessarilythesameas(thehandles)theupdatesthefilesystemis doing.And thatmeansthatbecausewearedoingwrite-behind,wemaybemakingonly onefilesystemcommiteveryfiveseconds,andyou canhavehundredsandhundredsof filesystemoperationsproceedingin thattimescale.Soeverythingcanbebatchedupveryefficiently usingthesehandles.[32m, 16s]
Now theAPI hasa journalstartanda journalstoppair. A journalstartgivesyouahandle.A journalstoptells thesystemthathandleis finishedwith. It providesnestedtransactions,soif you doa journalstartandthenanotherjournalstart,thatall getsbatchedinto thesametransaction.And thehandleis notmarkedascompleteuntilyou’vegonethroughtwo journalstopsin thatcase.[32m, 42s]
Whenyoudo thatjournalstart,youhave to tell theJFSlayerhow many blocksyouexpectmightbemodifiedby this update.That’s really, really important.This isabsolutelycritical, to avoid deadlockingin thejournal.TheJFSlayerhasto know
17
EXT3, Journaling Filesystem
(beforeit startscommittingyour transaction,sorry)beforeit startsprocessingyourtransactionthatthereis enoughspaceleft in thejournalto write out all of theblockswhich mightbecomepartof your transaction.[33m, 14s]
And if it turnsout there’s notenoughlog spaceleft, well it mightbethatyourtransactionincludesblocksfrom previoustransactionsin thejournal.And becausetheseblocksarenow beingpinnedaspartof a new transaction,wecan’t flushthemtodisk.And becausewecan’t flushthemto disk,wecan’t checkpointthelog to removeoneof theold transactionsfrom thelog. And becausewe’ve runoutof space,we’re justdeadlockedcompletely. [33m, 36s]
Sowehave to make thesekind of reservationsto makesurethetransactiondoesnotstartuntil all of thespacethatit might useis guaranteedpresentin thelog. Sothejournalstart/journalstopprovideboundariesto make thatsortof reservationguarantees.[33m, 50s]
There’s aquestionat theback.
[Thespeakercallsonanaudiencememberfor aquestion.]
Thequestionis: whathappensif there’s notenoughspacephysicallyin thejournalforthetransaction?Transactionsarevery limited in size;they’renevermorethana fewdozenblocks.Theonly two caseswherea transactioncangrow without boundsareforwrite systemcalls,becausewritescanactually... anapplicationcanquitelegitimatelywrite ahalf gigabyteof memoryto afile in onesyscall.[34m, 22s]
That’sokaybecauseEXT3 doesnotguaranteethatwrite is atomic.That’ll bebrokenup into multiple,smallertransactions.And theonly othercaseis truncate,becauseyoumight havea tengigabytefile thatyou’redeletingandyou really wantthatdeleteto beanatomicoperation,but thatdeletecantoucharbitraryamountsof diskspace.Potentiallyyouhaveoneseparatebitmapblock; in themostfragmentedcasein thefilesystem,you’d haveaseparatebitmapblockbeingupdatedon thedisk for everysingleblock thatyou freefrom thatfile. Sotruncatesarespecialcasewhich I’ ll cometolater, becausewehave to dealwith thatin anastymanner. [35m, 00s]
Actually, askany of thejournalingfilesystempeoplein theaudience:what’s thehideouspartof theentiresystem.And it’ sdeletingfiles.Everythingto do with deleteishairy. Everythingto dowith delete...youhavenightmaresaroundwhathappensif
18
EXT3, Journaling Filesystem
blocksgetdeletedandthenreallocated.[35m, 22s]
Whathappensif they getreallocatedwith adifferenttypeof data?Whathappensif youreallocatethemandthentakeacrashandhave to undothereallocationandundothedeleteoperation?And they really getnasty. [35m, 33s]
Somostof theproblemsin EXT3, mostof thehairy partsof thedesign,asin mostjournalingfilesystems,in fact,comefrom deletes.All of theblock updatesthatthefilesystemmakesafterthis timego througha pairof (systemcalls,sorry)journalingcallsexportedby theJFS.You canaskthejournalto giveyou write accessto ablockandrememberI saidearlierthatto ensureefficiency in thesystem,wedocopy-on-write.If there’sablock (beingjournalled,sorry)beingscheduledfor commit,thenwedon’t mind thefilesystemcontinuingto modify thatblockaslongaswehaveachanceto copy it outandmakesurethattheold snapshotthatwe’recommittingis stillconsistent.[36m, 14s]
And to achieve thatcopy-on-write,wehave to know whatthefilesystemis goingtomodify beforeit modifiesit. Soyouhave to go throughaprocessof gettingpermissionto write ablock, just to makesurethatcopy-on-writecanhappen.And thenat theendof it, wecansay:okay, thatblock hasnow beenfinished,it’ snow beendirtied; it cannow bewritten to a journalor whatever. [36m, 33s]
ThisAPI providesthereservation,it provides...thehandlesyou getbackfrom thisjournaling,youcanmarkindividualhandlesasbeingsynchronousbeforethejournalstop.And whenthefinal journalstophappensandthattransaction(is committedto thedisk,sorry)is bundledup for thecurrenttransactionon disk, it will immediatelysubmitthatcommitto thedisk,andwill synchronouslywait for thatcommitto finish ondiskbeforereturning.Soyoucangetsynchronousoperationonaper-handlebasis.[37m,03s]
And it alsoprovidesvarious...theJFSlayerprovidesvariouskind of managementfunctionsfor creatinga journal,doingrecoveryof a journal,markinga journalasbeingcomplete,flushinga journalout to disk, thingslike that.You needthatinsideEXT3.For example,if anEXT3 filesystemthat’s read-writegetsmountedread-only. If youremountit read-only, youwantto makesurethat,beyondthatpoint,no moreupdateshappento thefilesystem.Youwantto makesurethatnothingelsehappensin thejournal.You alsowantto makesurethatthere’s no recoverynecessaryon that
19
EXT3, Journaling Filesystem
filesystem.If you takea rebootwhile thefilesystemis markedread-only. Soall of theconsistency functionsnecessaryto flusheverythingoutandstopfurtherwriteswhenyou getoneof theseremounts.All of thesekindsof functionsareexportedby theJFSlayer. [37m, 52s]
Theimportantthing aboutthis API is thatall of theupdatesthatwe’re makingareexpressedin termsof: hereis a block,hereis thenew contentsof thatbuffer ondisk.We’re doingphysicaljournaling.Now there’ssomefilesystemsthatdo journalingwhich do logical journalingandthatmeansthat,for example,if they allocateaparticulardiskblock to aparticularfile, they’ ll write anentryin thejournalthatsays:this diskblock (is markedfree,sorry)is markedin use.And thisfile hasa mappingpointerpointingto thatdiskblock.And thatdescriptionfor thatallocationmight onlybea few byteslong.EXT3 will journaltheentirecopy of theblocksthathavebeenmodified,all 1K, 4K, whatever, of theblocks.[38m, 43s]
But it doesn’t doany copy whenit’ sdoingthat,sotheCPUoverheadis minimalandinparticular, if youhavegota lot of operationsthattouchthesameblocks,thatblockonlygetsjournalledonce.And thatmeansthatthis is avery, verysimplemechanismwhichgivesyou completelyfreecompressionof multipleevents.So,for example,youalmostnevergetadiskblock allocatedin a singlecommitoperation.If thecommitsaregoingeveryfiveseconds,thenif you only hadoneblockallocatedwithin thatfivesecondboundary, thenit’ sno big dealto write out4K; to markthatblock in use.But, asismorelikely, you’vegothundredsandhundredsof allocationsgoingin rapidsuccession.[39m, 24s]
All of thebitmapblocks,all of thebits in thebitmapswhicharebeingupdatedonlyresultin onecopy of thenew block beingsentto disk.Soit’ sactuallya fairly efficientwayof compressingthesekind of multipleoperations...multipledirectoryoperationsor multipleallocationswithin asingleblock group;they canall becompressedfairlyefficiently down to thedisk.Therearedifferenttypesof buffer thatyoucanpassthroughinto theJFSlayer. Now this is gettinginto theissueabout:arewe journalingeveryoperationon thedisk,or areweonly journalingmetadata?Rightnow, thecurrentversionof EXT3 that’s in public release:EXT3 is (journalingall metadata,sorry)journalingmetadataanddata.Soeveryfile thatyouwrite is beingwritten twice; it’ sbeingwrittenonceto thelog andonceto themaindisk.Thedesigngoalis obviouslynot to do that.[40m, 19s]
20
EXT3, Journaling Filesystem
Sothedesignis thatyoucanhavemetadatabeingjournalled,but thedataitself justgetswrittenbackto thediskany old way. Now if youdo that,awholepile of new orderingconstraintscomein. And guesswhat...they all have to do with delete.Exceptfor one.[40m, 43s]
Theonly onethatdoesn’t have to dowith deleteis: whathappensif youallocateawholepile of data,write it to disk,andthentakeacrash?Well, if you takeacrashandthenrecover thestateof thatjournalandreplayall of theallocationsto disk,but thedatahasnotyet beenwritten to disk, thentheusercanreadthosedisk locksandgetold, stalecontentsof whatwaspreviously thereon thedisk.And thatmight beanoldcopy of /etc/passwdor someotherfile you don’t really wanttheuserpokingaroundin.[41m, 11s]
Sofor security, you reallywantto makesurethatnewly-allocateddatablocks(will getcommitted,sorry)will getflushedto thediskbeforethetransactionwhich allocatesthemis allowedto commit.Sowehave this conceptof datablocksandthatwriteorderingguaranteefor datablocksis preservedby theJFSlayer. [41m, 30s]
Thenthereareawholepile of otherthingsthatcangowrong.Well, I canhaveadirectorywhich is deleted;I candeleteadirectoryandthatis aperfectlylegitimateoperationin EXT2. And I cando thatandI cancommitthatdeleteto diskandthenIcanhaveanew transactionwhich reallocatesthosesamedatablocks,whicharenowfree,andputstheminto afile. And I cancommitthat.And everythingis nowconsistent.[41m, 58s]
Exceptwhathappensif I takea reboot?Well whathappensis thatwedoa rebootandthelog getsreplayedandwego throughthejournal,replayall themetadatablocksthatarein thatlog. Well, actually, this directorythatwedeleteda few transactionsback,wemadeanew entryin thatdirectoryandthere’s ametadatablock for thatdirectoryentryon thejournal,but wedon’t journaldata.Sothatthenew contentsof thatdatablock,which areon themaindisk,aren’t in thejournal.And sowe takeacrashandwereplayandweoverwritethatdatablockwith theold contentsof thedirectoryblock,becausethat’sall that’s in thejournal.This is generallyconsideredbadbehavior. [Theaudiencelaughs.][42m, 40s]
Soweneedto havewaysof dealingwith that,soweneedto beableto haverevokerecords.Therearevariousdifferentwaysof dealingwith this; youcandealwith it by
21
EXT3, Journaling Filesystem
makingcertaindeleteoperationssynchronous,you cando it by makingsurethatyoudon’t reusediskblocksuntil you’vecheckpointedthatold recordoutof thejournal.Thewaywe’re doingthatin EXT3 is thatdeletingmetadatacancausea revoke recordto bewritten into thejournal.And whenyoudo thereplayof thejournal,theveryfirstpassof thejournalrecovery, we look for all of therevokerecordsandmakesurethatany datathat’sbeenrevokedis never, ever replayed.And sothatdealswith thatparticularcase.[43m, 20s]
Unfortunately, actuallywehadthoughtweweregoingto reallocatethatdirectoryasdata,we’vealreadyflushedthedatato diskandoverwrittenthedirectorythatwe’veactuallyjusthadto undelete.Whoops.Sowehave to makesurethatweavoid thatkindof thing.And thereare,again,variouswaysof doingthat.Theway thatthis worksinEXT3, is thatthejournalinglayerprovidesthefilesystemwith theability to recordthelastcommittedstateof thebitmapblock.And thefilesystemcanthenusethatto makesurethat,whenit is allocatingdata,it never, ever reusesthedatablock thathasbeenfreedin thebitmapsbut thatfreeinghasn’t yetbeencommitted.[44m, 49s]
Sothereareall theselittle tricky thingsthathappenarounddelete.And all of thewriteorderingconstraintstheJFSlayerhasenoughinfrastructurein thereto supportto thefilesystemfor gettingthis thing right. But theJFSlayerdoesn’t understandanythingaboutthedifferencebetweenthesedifferenttypesof data,it just hasorderingguaranteeswhich thefilesystemcanuseto makesurethatmetadata-onlyjournalingdoesactuallywork right. Thatis actuallyall implemented,but thecurrentEXT3doesn’t useit. It’ snotenabledfor thesimplereasonthatthesewrite orderingconstraintsarereally tricky to getright. And thepriority hasbeento makesurethatthecore,restof theEXT3 codeis rocksolid,beforewe startintroducingall of theseweirdandwonderfulnew bits andpieces.[45m, 28s]
22
EXT3, Journaling Filesystem
A few otherthingsthatwehave in theJFSlayer, there.It supportsnestedtransactions.Now this is whatweweresayingearlier, thatif youhave journalstartandyou getanotheroperationwhich doesa journalstart,well you cando thatquitehappily. Thenestedtransactionsallow you...Thiswasreally implementedfor quotafiles.[45m, 50s]
Now if anyone’s familiarwith theLinux VFSthere,they’ ll realizethequotasystemthatEXT2 usesis not insidetheEXT2 filesystem,it’ sactuallyagenericquotautilitylayerin theLinux VFS.And theLinux VFS(has)exportsfunctionsthatthefilesystemscanuse,sothefilesystemcansay:okay, I’veallocatedadiskblock;pleaseupdatethisquotarecordaccordingly. And theLinux VFSwill say:okay, fine, there’senoughquotaleft for you to do that;that’squitelegal; pleasego aheadwith thatandI’ ll makesurethattheappropriatequotarecordon somearbitraryfile is updated.And asfar astheVFS layer’sconcerned,thequotais just a regularfile. And it makeswrite callsto thefilesystemto updatethequotafile. [46m, 38s]
Now if youwantto makequotaupdatesconsistentwith therestof thetransactions,youactuallyneedthosewrite callsthatVFS is makingto bepartof thesametransactionthattheallocationon diskwaspartof. You have to makesurethey’reconsistent;youhave to makesurethatthequotafile updateis alwayspartof thesametransactionastheallocationthat’s (doingthat)modifying it. And by usingnestedtransactions,this comesasfree.[47m, 03s]
Thewrite systemcall startsthetransaction.It callsthequotalayer;thequotalayerdoesanotherwrite. Thatnew write for thequotafile startsa transaction,but it getsanestedtransactioninsidethefirst one,modifiesthequotafile, completesthetransaction.Andonly whenthewholeoperationis finisheddo wemarkthatasbeingacompletetransactionon thedisk.Sothisquotafile stuff, it just camefor free.And it turnsoutthatthis is actuallyquiteuseful.Thereareapplicationsout therethatwantto beabletousenestedtransactionsin thefilesystem.[47m, 31s]
And onein particularthat’salreadyusingthis is theInterMezzofilesystemthatTedtalkedaboutearlier. It’ sadistributedfilesystemthathasa local diskcachefor cachingfileson theharddisk locally. And there’s acoherency protocolthatthevariousdifferentnodesin this InterMezzonetwork useto communicatewith eachotherto say:okay, Ihaveanupdated...I’vegotanewercopy of this particularfile; I’ ll propagateit to all theotherservers;I’m aboutto modify this copy of thefile, soI’m goingto invalidatethe
If I havepopulatedmy local diskcachewith copiesof themainserver’s in InterMezzo,thenI canput thatonmy laptopandtake it away, go to this conference,haveall of mye-mailon that;haveall of my otherstuff on that;andthenI getbackhome,I canplug itinto thenetwork andeverythinggetsreplayed.And thedisconnectedoperationis reallypowerful. [48m, 29s]
But thatmeansthatthelocaldiskcache...therehasto beawayof recordingwhathaschangedin thelocaldiskcachesothatwhenwedoreconnectto therestof thenetwork,wecanreplaywhat’schanged.And InterMezzoactuallyusesEXT3 for this, to getvery, veryhighperformancein thelocal diskcache.Becauseit createsanestedtransactionandsayI’m runninga disconnectedoperationandI’m makinganew file onmy filesystem.InterMezzowill createthatnew file in its diskcacheandwill updateajournalfile thatInterMezzomaintainsto say:this inodeoverherehasbeencreatedintheInterMezzofilesystemwith such-and-sucha filenameandit belongsto thisparticularplacein thecache.[49m, 13s]
And InterMezzoreally wantsto makesurethatthecontentsof its log, of its replaylog,matchwhat’sactuallyon thefilesystemcache.And it canuseEXT3 for that;it canuseanestedtransaction.It will startanestedtransaction,createthefile, write anupdaterecordto its replaylog statingthatfile hasbeencreated.And it getsnormalwrite-behindsothere’sno synchronousupdatesondisk. It’ s fully asynchronouswrite-behindperformanceto thatlocal diskcachebut afteracrash,it guaranteesthatthatreplaylog is exactlyconsistentwith theactualstateof thediskcache.SoInterMezzois usingthesenestedtransactions;it’ sprovenvery, veryusefulfor them.[49m, 52s]
Now wegetto someof theseotherawkwardlittle things.Orphanedfiles,asI saidearlier, if wehaveafile which hasbeenunlinkedondisk,but is still open,thenon thereboot,weneedto makesurethatfile is deleted.EXT3 addsanew datastructureon thedisk. It hasanentryin thesuperblockwhich pointsto a linkedlist of inodeson diskwhich needto bedeletedon reboot.And wheneveryou unlink anopenfile, it getsaddedon to thatlist. And whenyou finally closethatfile, thedeleteoperationwhichhappensasa resultof thatclosewill removetheinodefrom thatlist. [50m, 37s]
24
EXT3, Journaling Filesystem
I saidearlierthattruncateoperationsalsocanhaveunboundedtransactionsizes.Wellthat’sokay;if yougeta truncateoperationwhich exhauststhesizeof thejournal,thenthattruncatewill besplit up into morethanonetransaction.But westill guaranteethatthat’satomicovera reboot,becauseif wehave to split up oneof thesetruncatesovermultiple transactions,weput it onexactly thesameorphanedfile list. And soinrecovery, youcandoall thecleanup.It basicallymeansthatwhenyou dorecovery, welook at thenumberof links, thenumberof hardlinks for thatfile in theinodelist, andifthenumberof links is zero,weknow it’ safile that’s just beendeletedandwedeleteit.Wefinish doingthedelete.if thenumberof links is greaterthanzero,weknow it musthavebeenin themiddleof a truncate,andsowe completethetruncateoperation.Soallof thatkind of getsdoneat theEXT3 level; thatcannotbedoneinsidethejournalinglevel. [51m, 33s]
VM reservations...journalinghasthis unfortunatepropertythatif youhaveatransactionthat’s in progress,youcannotfreethememorythattransactionis occupyingwithout first allowing thattransactionto complete.Becauseunlikedatabases,mostfilesystemsdo not implementtransactionaborts.It’ s just not somethingwhich isnormallyneededin a filesystem.And if you’renotgoingto beableto abortatransaction,thenyouhave to let thetransactionsrun to completion.And if we’re goingto let thetransactionsrun to completion,youhave to haveenoughmemoryto do that.And if theVM systemis saying:well I can’t giveyouany morememoryright nowuntil yougivemebacksome;thenyoucandeadlockvery, very rapidly. [52m, 18s]
Soweneedto havea wayof doingVM, virtual memory, (transactionreservations,sorry)pagereservations,sothatthefilesystemdoesn’t usemorememorythantheVMlayeris ableto givebackindependently. And it turnsout thatthis is a relatively trivialthing to addto theEXT3 filesystem,becausejournalingfilesystemsalreadydo thesamekind of reservationin thelog. Sooneof thethingsthattheVM developersandthejournalingfilesystemdevelopershavebeentalkingaboutrecently, is how to addanAPIto thevirtual memorylayerwhich allows thefilesystemsto tell theVM aboutthereservationsit’ smakingandto makesurethatwenever run into thesedeadlocksituations.[52m, 57s]
Therearealsosometricky cornersaboutwrite pressure.For example,if you’vefilledtoomuchmemorywith dirty data,andit’ sproving impossibleto clearstuff outofmemory, becauseall of thepagesin memoryaredirty andneedto bewritten to disk
25
EXT3, Journaling Filesystem
first, thenweneedto, at thatpoint,stopmakingmoredirty data.Wealsoneedto startcleaningtheexistingdirty pagesbackto disk.Now thefilesystemhasits own writeorderingconstraints.It cannotlet theVM arbitrarilydecideto write thesethingsbackto disk.[53m, 33s]
But theVM is theonly partof thesystemthatknowswhenthis write pressureis gettingexcessive.Sowe have to havecallbacksinto thejournalingfilesystemwhich let theVM say:well, hey, you’vegotall thesedirty pages,I wantto startwriting themback.But thefilesystemhasgot to beableto say:well, actuallythesedirty pagesarepinnedin memorybecauseof transactions,but I’vegotpagesoverherethatI canfree.Sowewantto havecallbacksfrom thevirtual memorysysteminto thefilesystem.Purelyadvisorycallbacks,sothattheVM cansay:I have foundall thesedirty pageswhich Iwantyou to getrid of, but if youcan’t do it, I don’t careaslongasyoufind somethingto getrid of. [54m, 10s]
And onceyou’vegot thatkind of advisorycallback,dumbnaivefilesystemslikeFAT orEXT2 cansayto theVM: okay, you told meto write this pagebackandI’ ll do it.Whereadvancedfilesystemsthatdo journalingcanusethatasanindicationthatweneedto dosomewrite pressureandcanchoosethemostappropriatepagesto getrid of,becausethey know whatthewrite orderingconstraintsare.Soall of theselittle tricksthatweneedto do betweentheVM andtheVFS...hopefullywe’ll geta lot of thatinbefore2.4 is released.And thatwill allow thecleanmergingof awholeclassofdifferentjournalledfilesystemsthathaveexactly thesameorderingconstraintsthattheVM isn’t awareof right now. [54m, 43s]
And thenfinally, we’vegot this write orderingconstraintat theSCSIlevel. Which ourmanin front wastalkingabout.Rightnow, theonly way thatwecandocommitssafely,is by waiting for thedisk to tell usthattheentiretransactionhashit disk.And only thenwill you give thecommitblock to thedisk.[55m, 08s]
[Thereis acommentfrom theaudience.]
“That’son media;not just on disk,becausethere’s lots of media.”
Yes,on media.Well if adiskhasgotwrite-behindbattery-backedcache,thenthat’sfine;wedon’t carewhatkind of media...aslongasit’ smadepersistent.As longasthelog updatescanbemadepersistent,thenwecanallow thecommitblock to go to disk.
26
EXT3, Journaling Filesystem
It turnsout thatSCSIhasaverynicefeaturecalledtaggedcommandqueueing.Youcanhaveawholepile of diskoperations,diskwrites,outstandingon thediskat once,andthediskcando themin any orderit cares.And it will just tell you thatit’ s finishedinwhateverorderit happensto choose.[55m, 47s]
Thereis abit in theSCSIcommandcalledorderedtag.Youcanactuallysetanorderedtagbit in thetaggedcommandqueueandthatis awrite barrierto theSCSIlayer. TheSCSIlayerwill guarantee,theSCSIdiskwill guarantee,thatnowritesthatyousubmittedbeforethatorderedoperationwill overtake it. And thatno writessubmittedafterit will besentto diskbeforethatoperation.And if wehave, in theblock deviceI/O layer, if wehaveawayof specifyingthatbarrieroperation,saying:this is acommitblock,don’t reorderthis block; reorderanything insideof it, but don’t reorderthisone...if wecansetthatin theblockdevice layersothattheLinux internaldevicereorderingqueuesandthedisk’s reorderingqueuesall observeandhonorthatbarrieroperation,thenwecankeepthepipelinegoingto thedisk,streamingthedatato thediskat full speed,withoutwaitingsynchronouslyfor thecompletionof thesetransactions.[56m, 59s]
And thatbecomesreally importantfor a few specialcases.Thatbecomesreallyimportantif you’resynchronouslycommittinga lot of fasttransactionsto disk.Andtherearecaseswherethatreallyhappens.Mail spoolsandNFSservers...thetwocanonicalexamples.Mail spoolsareconstantlyupdatinglots of smallfilesandtheywantto doanfsync()aftereachoneto makesurethey don’t tell thesenderthatthemailhasbeenreceiveduntil thediskhasrecordedthefactthatit’ s safeondisk.[57m, 25s]
And in thecaseof NFS,NFSv2or NFSv1serversareexpectednot to acknowledgethewrite to diskuntil it’ s safeondisk.Becausethewholepoint of NFSrecovery is that,iftheservercrashes,theclient will replayany of thecommandsthatweren’tacknowledgedby theserver. But anything thatwasacknowledgedby theserver isassumedto besafeondisk.And therefore,theserver... if that’s to work overa servercrash,theservercan’t acknowledgetheNFScommandsuntil it’ ssafeondisk,whichmeansthattheNFSserver is typically doinglargenumbersof verysmalldatawrites,synchronously. [58m, 00s]
And in EXT3, we wantto beableto have thosewritesspoolingsequentiallyto aseparatedisk,which is thejournaldisk.Rememberthatthejournalwill notnecessarily
27
EXT3, Journaling Filesystem
have to bein thesamediskasthefilesystem.Sowewantto just beableto spoolthisstuff sequentiallyontothejournaldiskat full speedandhaving to wait for all of thelogto bewrittenbeforeyousubmitthecommitrequest,thecommitI/O, typically meansthatyou’rewastinga wholerotationallatency in thedisk,wheneveryou’redoingacommit.If wecangetthesewrite barriersright throughtheLinux I/O layers,thatreally, really improvestheperformanceof thesestreamingsynchronousI/Os.That’sonething that’son thecards,it will probablynotgetinto 2.4,but theI/O layersinLinux will hopefullyhave this in 2.5.[58m, 46s]
I’m assumingthatthewrite orderingon thefirmwareof thedisk is functional,correct.Iassumethatwhenthedisksaysthatit’ swrittensomething...[Someonecommentsinbackground.]Ohyes,I know... Wehavegot,wewill only enablethis with eitherawhite list or ablacklistedsetof drives.We’vealreadygotdriveblacklistsin thekernelfor thingsthatlie hideously... Absolutely, thatwill notbeadefault behavior; wehave tobeverycarefulaboutthat.[59m, 27s]
Sothestatuson this.Thecoreof EXT3 is... theoneI’m runningon thelaptophereisabsolutelyrobust;thereareno known problemsin that.Someof thekind of userinterfaceissuesarenotquite100%.If youdeletethejournalor yougive it aninvalidjournalinodeor thingslike that,thentheactualsetupof thejournalingcansometimesgetabit confused,but thosearemanagementissuessurroundingtheuserlevel toolswhich arebeingusedto managethat.[59m, 56s]
Theuserlevel toolsis themainthing thatwe’re workingon right now. And themetadata-onlyjournaling.Theactualcorefilesystemis rocksolid. It’ s beingusedinproductionweb/FTPserversfor multiple tens-of-gigabytefilesystems;I trustitabsolutelywith thelaptophereandgiventhestatusof someof thedevicedriversthathavebeenrunningon this laptoprecently, it’ s ratherconvenientto have theability torebootvery, veryquickly. [Theaudiencelaughs.][60m, 25s]
Thee2fsprogshasat theminute...it hasminimal supportfor EXT3, but it is there.Itwill understandthepresenceof therecoverybits in thejournalbits andwill do
28
EXT3, Journaling Filesystem
appropriatethings,at leastin asmuchasit will not touchafilesystemthatit doesn’tknow how to touch.And it will not complainabouttheexistenceof a journalif thejournalhappensto bethere,but doesnotneedrecovery. [60m, 51s]
Sothereis ongoingwork for thingslike themetadata-onlyjournaling.Theinfrastructurein theJFSfor thatis all implementedexceptfor therevokerecords;thecurrentsuperblockformat,thejournalformat,doesnothaverevoke records.That’s theonly thing thatneedsto beaddedbeforeI canenableall of thatsupport.[61m, 13s]
Thequotastuff is in there.It hasnotbeenfully tested.I know therearepeopleusingit,but it’ snot testedasmuchastherestof thecode,soI needto havemoretestingdoneon thatbeforeI recommendit asbeingstable.[61m, 27s]
And thereareacoupleotherthingswhicharebeingworkedon.For example,I’m goingto bemoving thejournalfrom beinga regularfile to beinga reservedfile andhaveatune2fsfunctionwhich will allow you to addanarbitrarily sizedjournalinodeto anexistingEXT2 filesystem,without it appearingin thefilesystemnamespaceandtherefore(without it polluting,sorry)without it temptingpeopleto deleteit andnastythingslike that.[62m, 14s]
Sothat’sbasicallywherewe’reat right now. ThecurrentcoreEXT3 is stable;it doesjournaldataandthereforeit haspoorwrite performancefor write-intensiveoperations.But it is reliable,andit will run...it canbeaddedtransparentlyto any existingEXT2filesystem.Thatwill bemaintainedasastablebranchwhile I’m merging in themetadata-onlyjournaling.Sotherewill beadevelopmentbranch.This is all 2.2; the2.4port will only happenonceall this is stablebecauseit’ snot possibleto achieveasufficiently high level of reliability if you’rechangingtoomany thingsat onetime. Itjust makesit somuchharderto maintainanddebug.[63m, 00s]
Therearesomethingswhich I’m leaving until afterthefirst completestablerelease.And thatis, in particular, althoughtheJFSlayerunderstandsoff-disk journalingandunderstandsmultipleblockdevicessharingthesamejournal,thereareanumberof
29
EXT3, Journaling Filesystem
really nastymanagementissuesin termsof administrationof thatkind of environment.[63m, 23s]
Like,whathappensif you’vegot tenfilesystemssharingthesameoff-disk journalandon reboot,oneof thosefilesystemsdies?Well youcan’t actuallystartreusingthecontentsof your journaluntil you’vedoneall of therecovery. And you can’t do all oftherecoveryuntil you’vegotall of thefilesystemsmountedor at leastfinished...at leastuntil you’ve foundall of thefilesystemsthatusethatjournal.And if oneof thosefilesystemshasdisappeared,you can’t do recoveryontothatfilesystem,sothereforeyou can’t startre-usingthejournal.[63m, 52s]
Thereforeall of theotherfilesystemswhich aresharingthesamejournalgosouth.Soit’ sa little problem.Wehave to do thingslikemakingsurethattherecoverycodehastheability to untanglethedifferentbits of a journalinto aseparatefile andstorethatinthetempdirectorysomewhere,sothatwhenthatmissingfilesystemgetsfoundlateron,wecando therecovery then.[64m, 17s]
Thereareall sortsof little thingslike thatthatwehave to dealwith whenyou’vegotoff-disk journalsandsharedjournals;which just don’t evercomeinto thepictureifyou’re journalingon thesamedevice thefilesystemis on.Sothatwill probablybeapost1.0 issueandoneotherthing thatI wantto do is to actuallyexport thenestedtransactionAPI into userspace.Youhave to bevery, verycarefulaboutthatbecauseit’ snotpossibleto guaranteeproperdatabasesemantics.You can’t haveunbounded,largetransactions.Youhave to havesomeway in which theuserapplicationcangetinadvancesomeideaof how many diskblocksit’ sgoingto needto modify for theoperation,becauseit’ sgoingto call variousthingslike thatwhich arenotentirelystraightforward;it’ snotquiteassimpleaspeoplewouldhope.But it’ s sufficientlyusefulthatthatwill beexportedto userspaceat somepoint.[65m, 07s]
That’sall to solve for EXT3. EXT3 is not theonly thing that’sgoingon with EXT2.Thereareotherthingsthatarehappeningin theEXT2 filesystemspaceasseparatedevelopmentbranches;muchlikeEXT3 is adevelopmentbranchoff of EXT2. And it’ slikely thatsomenumberof thesewill bemergedinto asingle,new EXT2 variantsometimein thefuture.Therearepeopleincreasinglyhammeringfor securitysupport.Accesscontrollists,mandatoryaccesscontrollabels,capabilities...all thattypeofthing.Therearepeoplewho reallywantthatin Linux. [65m, 40s]
30
EXT3, Journaling Filesystem
Thereareproposalsfrom theUSDepartmentof Defensethatwill forbid themfrompurchasingany operatingsystemwhich doesnot have(thesefacilities)thesecapabilities.And they havebeentrying to persuadetheUSGovernmentto adoptthesamerules.Althoughthey areresistingthat.[66m, 00s]
There’s B-Treesupport.B-Treesarefairly complicatedon-diskstructures.Therearepeoplewhowantto haveB-Treesupport(in Linux) in EXT2, for scalabledirectoryperformance.But B-Treescangoreally horribly haywireif you interruptthemin themiddleof a treebalancingoperation.And somakingthemconsistentovera rebooteitherrequiresthatyou’revery, verycarefulanddo lots of extra I/O in thediskstructureto makesurethatit canberecoveredsanely, or youdo journaling.[66m, 26s]
In fact,there’s awholesectionof codeinsidereiserfswhich wasdealingwith exactlythis issue.Which hasbasicallybeeneliminatedfrom thefilesystemnow thatthey’vegot journaling,becausejournalingdealswith all thatfor them.PuttingB-TreesintoEXT3 really requiresJFSasaprerequisite.[66m, 47s]
Onlineresizeis beingdonefor EXT2. And thereareanumberof otherminimallyintrusiveextensionsthat...EXT2 hasthis advantageof beinga verynice,simplefilesystem.And youcando B-Treeextentmapsinsideyour filesasaveryefficientwayof encodinglargefilesonyour disk.Now youdon’t actuallyhave to useB-Treesforthat.If youdon’t useB-Trees,you canjust havea verysimpleextentmapstructureonthedisk,whichmapsentirecontiguousextentsof on-diskblocksfor asinglefile in justa few bytesin thedirectorystructuresin theinode.[67m, 23s]
Thatis all well andgooduntil youstartdoingthingslikehaving holesin thefilesandthenwantingto write into themiddleof thoseholes.Onceyou do that,thenyou’rehaving to shuffle aroundall of your extentmapsandsoon.And thatgetsreallycomplicated;that’snormallywhy peoplewantB-Treesfor extentmaps.If you’rewilling to foregowith theability to write into holesinto yourfilesystem,youdon’tneedB-Treesfor yourextentmaps.[67m, 48s]
Sothat’ssomethingthatwecoulddo in EXT2 asanexperimentto say:what’s theminimalnecessarymodificationto thisfilesystemto provideall of thebenefitsof extentmaps,exceptfor this facility with holes;whicharegoingto beusedby avanishinglysmallfractionof users.Wedon’t needto do B-Treedirectories;wecanmaybedohashingof thedirectories;averymuchsimplerextensionto thefilesystem,whichcan
31
EXT3, Journaling Filesystem
giveusmany of theperformanceimprovementsof B-Treedirectories,but without theimplementationcosts.Soreally therearelots of thesethingsthatpeoplearelooking atwith EXT2. [68m, 24s]
No, I’ve told you there’sgoingto bea tune2fsoptionswhich doesall of thatfor youandputsit into a reservedinode;that’ll all beinvisible.[69m, 31s]
[Thereis anadditionalquestionabouthaving to diskseekto thejournalevery timeyoudo awrite; aproblemsimilar to thosewith FAT filesystemsthatneedto write to theFAT for everydiskwrite.]
Whataboutcrashduringrecovery?It doesn’t matter. Recovery just consistsof goingthroughthelog andwriting what’s in thelog backto disk.And writing it backto disktwice is just asgoodaswriting it once.Soif youcrashduringtherecoveryandyou’vewrittenhalf of thelog, well thenext recoverygoesalongandjust doesall thesamesetof writes,plusa few more.And only whentherecovery is completed,do wemake themodificationto thejournalwhichmarksrecoveryasbeingdone.Therearenomodificationsto thejournalwhile recovery is in progress.Soit just works.There’s noproblemsthere.[72m, 32s]
[Thereis anotherquestionfrom theaudience.]
Doesthis at themomentwork with softwareRAID? It will work with hardwareRAID;it will work with softwareRAID oncethe2.4port is done,becauseabug in softwareRAID hasbeenfixedin 2.4.It will not work with softwareRAID in 2.2.Thereasonforthatis thatsoftwareRAID whenit doesRAID recoveryafteracrashworksby doingabuffer cacheread,stripe-by-stripe,throughthewholedisk,writing thosestripesback.And whenit writesthosestripesback,it updatesthediskswhich werenotconsistentatthetimeof thecrash.Unfortunately, whenit’ sdoingthat,it’ s causingthecontentsofthebuffer cacheto bewritten to diskwithout thefilesystem’ssay-so.And thereforeis
