Dark Side of Git - We use this on my big data team

transcript

How GIT Works InternallySeongJae Park <sj38.park@gmail.com>

Nice To Meet You

SeongJae Park

sj38.park@gmail.com

DVCS(Distributed Version Control System)http://git-scm.com/images/logos/downloads/Git-Logo-2Color.png

DVCS(Distributed Version Control System)

Made-by Linus Torvalds For Linux

http://git-scm.com/images/logos/downloads/Git-Logo-2Color.png

http://cdn.memegenerator.net/instances/400x/37078331.jpg

Many Projects Use Git Because It’s Awesome

http://blog.appliedis.com/wp-content/uploads/2013/11/android1.pnghttp://upload.wikimedia.org/wikipedia/en/4/40/Octocat,_a_Mascot_of_Github.jpghttp://upload.wikimedia.org/wikipedia/commons/thumb/3/35/Tux.svg/512px-Tux.svg.png

Hard To LearnConfusing For CVCS UsersPush? Pull? Fetch? Rebase? HEAD???

http://www.quickmeme.com/img/fd/fd09e17b3393b2ea1cd7e52af1ad7c77f3c2d7a83e9f47d4b90ba3af52dde329.jpg

Git: The Information Manager From Hell

http://www.youblob.com/sites/default/files/styles/large/public/field/image/frontlego1.png?itok=XA5CXt84

$ git log e83c516

commit e83c5163316f89bfbde7d9ab23ca2e25604af290

Author: Linus Torvalds <torvalds@ppc970.osdl.org>

Date: Thu Apr 7 15:13:13 2005 -0700

Initial revision of "git", the information manager from hell

That’s Why So Confusing And Hard To Learn

$ git log e83c516

commit e83c5163316f89bfbde7d9ab23ca2e25604af290

Author: Linus Torvalds <torvalds@ppc970.osdl.org>

Date: Thu Apr 7 15:13:13 2005 -0700

Initial revision of "git", the information manager from hell

This Time, We Will...

See How Git Works From The Scratch

https://lh4.googleusercontent.com/gBpfuABUjSNi2RagtJrGi8TW-pmtgak_0qtGOGubihvKH-5-umreO9CwJgjX2kaA9E7RkLwtEwiDnoMtOgm4iMJ0IWhvXlzlKL1kNVUYWuNa-gLRtRoyNjkVYg

Just For Fun...Or To Be Friend Of Git

Forget About TheComplicated CommandsThis Time

In Short,

Git Is A Content-Addressable Storage System

http://www.juliagiff.com/wp-content/uploads/2014/03/tldr_trollcat.jpg

In Short,

Git Is A Content-Addressable Storage System

Blob, Tree, Commit, Reference. That’s It =3

Plumbers: Unsung Heroes Behind

● Git Looks Graceful Owing To Plumbing Commands Consisting Them

http://cfile4.uf.tistory.com/image/182FF7244CFDDFB33CC999http://cfile29.uf.tistory.com/image/18574F224CFDD89B163073

Plumbers: Unsung Heroes Behind

● Git Looks Graceful Owing To Plumbing Commands Consisting Them○ The Wounded Foots Are What We Interested In

http://cfile4.uf.tistory.com/image/182FF7244CFDDFB33CC999http://cfile29.uf.tistory.com/image/18574F224CFDD89B163073

Again, From The ScratchVCS? Why? How?

Why VCS?

Usual Life Of File

FileA ver 0 FileB ver 0

Why VCS?

Usual Life Of File

FileA ver 0 FileB ver 1FileB ver 0

Why VCS?

Usual Life Of File

FileA ver 0 FileB ver 1

Why VCS?

Usual Life Of File

FileB ver 1 FileA ver 1FileA ver 0

Why VCS?

Usual Life Of File

FileB ver 1 FileA ver 1

Why VCS?

Usual Life Of File

FileB ver 2FileA ver 1FileB ver 1

Why VCS?

Usual Life Of File

FileB ver 2FileA ver 1

Why VCS?

Usual Life Of File

FileB ver 2FileA ver 1

We Need Version Control System

VCS Would...Record Every Changes Safely, Efficiently

VCS Would...Record Every Changes Safely, EfficientlyAble To Check Out Any Version

VCS Would...Record Every Changes Safely, EfficientlyAble To Check Out Any VersionEasy To Read History

Brute-force IdeaVersion Control Using File System

Brute-force Idea

Rename / Backup Every Files Whenever Change Made

Brute-force Idea

foo_20140111.c

Brute-force Idea

foo_20140111.c

foo_final.c

Brute-force Idea

foo_20140111.c

foo_final.c

foo_realfinal.c

foo_planb.c

foo_finalfinal.c

Brute-force Idea

foo_20140111.c

foo_final.c

foo_realfinal.c

foo_planb.c

foo_finalfinal.c

Brute-force Idea + History Isolation

Keep Working / History Directory Seperately.

Brute-force Idea + History Isolation

Keep Working / History Directory Seperately.Better, But...

$ find . -type f

./working/foo.c

./history/foo_20140111.c

./history/foo_final.c

./history/foo_realfinal.c

./history/foo_planb.c

./history/foo_finalfinal.c

TODOs From Version Control Using FS

Use Storage Space-Efficiently

Use Storage Space-EfficientlyEasy History Searching

Mission #1:Store History Space-Efficiently

Basic Idea: Avoid Duplicated Objects

Content-Addressable Storage System

Content-Addressable Storage SystemKey: SHA-1 Hash Of Object’s Content

Value: Compressed Content

Content-Addressable Storage SystemKey: SHA-1 Hash Of Object’s Content

Value: Compressed Content

Same Content Never Saved Twice

Save / Load ‘homer’

$ mkdir simpsons; cd simpsons; git init

Initialized empty Git repository in simpsons/.git/

$ echo ‘homer’ | git hash-object -w --stdin

de7e45490c9a4a3b5d5fae106faa4235ec669e02

$ find .git/objects/ -type f.git/objects/de/7e45490c9a4a3b5d5fae106faa4235ec669e02

$ git cat-file -p de7e4

keep it simple, stupid

$ git cat-file -t de7e4

What `hash-object -w` did

hash_object_w(‘homer\n’)

# Save compressed header + content at sha1 path

def hash_object_w(content):

header = ‘blob %d\0’ % len(content)

store = header + content

sha1 = sha.new(store).hexdigest()

dir = ‘.git/objects/’ + sha1[0:2] + ‘/’

filename = sha1[2:]

dir = ‘.git/objects/’ + sha1[0:2] + ‘/’

filename = sha1[2:]

open(dir + filename, ‘w’).write(

zlib.compress(store))

Version Control Using Hash Value

$ echo “bart” > son

$ git hash-object -w son

e00ddae83bdab443f4267426623aa34636c935f2

$ echo “hugo” > son

8e1e2f09585e021c9727585af72e10871d7be7ce

e00ddae83bdab443f4267426623aa34636c935f2

$ echo “hugo” > son

8e1e2f09585e021c9727585af72e10871d7be7ce

# Need former version, “bart”

$ git cat-file -p e00dd > son

$ cat son

Use Storage Space-EfficientlyEasy History Searching

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History

https://www.sciencenews.org/sites/default/files/main/articles/sad_opener.jpg

● TODO○ Support Directory Structure○ History Management○ Better Reference Than Hash Value

Q: What If Small Changes Inside A Big File?

$ du -h bigfile.c188Kbigfile.c$ du -sh408K.$ echo ‘/* small change */’ >> bigfile.c$ git commit -as -m “small change, big difference”$ du -sh496K.$

Q: What If Small Change Inside A Big File?A: Git Pick up Diff-Only If Necessary

But, Don’t Forget To Keep It Small, Simple

$ du -sh496K.$ git gc

Counting objects: 6, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (4/4), done.

Writing objects: 100% (6/6), done.

Total 6 (delta 1), reused 0 (delta 0)

$ du -sh

Mission #2:Store History Of Directories

tree Object

Point Other Objects(Using Hash) With Name

tree Object

blob blob tree

a113f2mommy b8934

c9240pets

d9b13cat

tree Object

“A Root tree Object Is A Snapshot”

blob blob tree

a113f2mommy b8934

c9240pets

d9b13cat

I’m a snapshot

tree object$ mkdir pets; echo ‘snowball’ > pets/cat

$ git update-index --add son pets/cat

$ git write-tree

15ee76ed3e744b6796950d07f26283d033ea3ea7$

$ git write-tree

15ee76ed3e744b6796950d07f26283d033ea3ea7$ git cat-file -p 15ee7

040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob 8e1e2f09585e021c9727585af72e10871d7be7ce son

$ git write-tree

15ee76ed3e744b6796950d07f26283d033ea3ea7$ git cat-file -p 15ee7

040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob 8e1e2f09585e021c9727585af72e10871d7be7ce son

$ git cat-file -p 85ab7

100644 blob 6a1f952e1baedcb3db93a3ea5e3389e5a87941e9 cat

$ git cat-file -p 6a1f9

snowball

Internal Data Structure

blob tree

8e1e2son

85ab7pets

blob tree

6a1f9cat

8e1e2son

85ab7pets

Version Control Using tree Object

$ git update-index --add son

$ git write-tree

661e6ad514a7f05c46c2931280cb78a339d34ee2

$ git write-tree

661e6ad514a7f05c46c2931280cb78a339d34ee2

$ git cat-file -p 661e6040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob e00ddae83bdab443f4267426623aa34636c935f2 son

$ git write-tree

661e6ad514a7f05c46c2931280cb78a339d34ee2

$ git cat-file -p 661e6040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob e00ddae83bdab443f4267426623aa34636c935f2 son

$ git cat-file -p e00dd

blob tree

8e1e2son

85ab7pets

6a1f9cat

blob tree

e00ddson85ab7

8e1e2son

85ab7pets

6a1f9cat

● TODO○ Support Directory Structure○ History Management○ Better Reference Than Hash Value

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure

● TODO○ History Management○ Better Reference Than Hash Value

Mission #3:Commit Message

commit Object

Describe Who / When / Why The Change Made

http://modthink.com/wp-content/uploads/2013/05/WhoWhatWhenWhereWHY.jpg

commit Object

Describe Who / When / Why The Change Made

Point A tree Object With Information Above

http://modthink.com/wp-content/uploads/2013/05/WhoWhatWhenWhereWHY.jpg

commit Object

$ echo '1st commit' | git commit-tree 661e6

0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0

commit Object

$ git cat-file -p d075ctree 661e6ad514a7f05c46c2931280cb78a339d34ee2author SeongJae Park <s**@gmail.com> 1410527921 +0900

committer SeongJae Park <s**@gmail.com> 1410527921 +0900

1st commit

commit Object

$ git cat-file -p d075ctree 661e6ad514a7f05c46c2931280cb78a339d34ee2author SeongJae Park <s**@gmail.com> 1410527921 +0900

1st commit

Who WhenWhy

Version Control Using commit Object

$ echo '2nd commit' | git commit-tree 15ee7 -p 0ca73

003b5e66caa89a6228c7b4d91e0475e56bf1bdf6

$ git cat-file -p 003b5

tree 15ee76ed3e744b6796950d07f26283d033ea3ea7

parent 0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0author SeongJae Park <s**@gmail.com> 1410528231 +0900

2nd commit

That’s Why People Says, “A Commit is a snapshot”

blob tree

commit commit

parent

85ab7pets

8e1e2son

85ab7pets

6a1f9cat

e00ddson

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure

● TODO○ History Management○ Better Reference Than Hash Value

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure○ Manage History Well

● TODO○ Better Reference Than Hash Value

Mission #4:Human Readable Name

Git References

File With Human-Readable Name

Git References

Storing SHA-1 Value Of commit Object

Git References

Storing SHA-1 Value Of commit Object

Resides In .git/refs/

Git References Using echo

$ echo "0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0" > .git/refs/heads/first

$ git log --pretty=oneline first

0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0 1st commit

$ git log --pretty=oneline first

$ find .git/refs/heads -type f

.git/refs/heads/first

.git/refs/heads/master

Git References Using update-ref

$ git update-ref refs/heads/master 003b5

$ git log --pretty=oneline master003b5e66caa89a6228c7b4d91e0475e56bf1bdf6 2nd commit

$ cat .git/refs/heads/master

003b5e66caa89a6228c7b4d91e0475e56bf1bdf6

blob tree

commit commit

parent

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat

blob tree

commit commit

parent

refs/heads/master

refs/heads/first

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure○ Manage History Well

● TODO○ Better Reference Than Hash Value

Version Control Using Reference

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure○ Manage History Well○ Easy To Remember Specific Snapshot

● TODO○ ...cooperation?

FAQ #1How Git Make-up Working Directory?

How Git Knows Current Commit?

Answer: HEAD

HEAD Points reference Using ref format(Not SHA-1)

Answer: HEAD

HEAD Points reference Using ref format(Not SHA-1)

$ cat .git/HEADref: refs/heads/master

HEAD$ cat .git/HEAD

ref: refs/heads/master

HEAD$ cat .git/HEAD

$ git branch

* master

HEAD$ cat .git/HEAD

$ git branch

* master

$ git symbolic-ref HEAD refs/heads/first

$ cat .git/HEAD

ref: refs/heads/first

$ git branch

* first

master

blob tree

commit commit

parent

refs/heads/master

refs/heads/first

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat

blob tree

commit commit

parent

refs/heads/master

refs/heads/first .git/HEAD

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat

FAQ #2Cloned. Now Fetch Or Pull ?

Fetch / Pull

Fetch Or Pull To Get Latest Code?

● Just Fetch Remote Repository’s Objects And References To Local Git Internal Storage

● If You Need The Changes On Your Working Directory,

● Just Fetch Remote Repository’s Objects And References To Local Git Internal Storage

● If You Need The Changes On Your Working Directory,○ Manually Merge Them Using git-merge Or,○ Checkout

Refspec Describes Source / Destination

$ cat .git/config | grep remote -A3

[remote "origin"]

url = git://10.0.0.1/git/simpsons.git

fetch = +refs/heads/*:refs/remotes/origin/*

Source Destination

Fetch: Beforeurl = git://10.0.0.1/git/simpsons.git

blob tree

a134fson

799cfpets

7cc07cat

65464son

799cfpets

commit commit

parent

refs/heads/master

.git/HEAD

git://10.0.0.1/git/simpsons.git

blob tree

a134fson

799cfpets

7cc07cat

commit

refs/heads/master

.git/HEAD

file:///home/sjpark/simpsons

Fetch: Afterurl = git://10.0.0.1/git/simpsons.git

blob tree

a134fson

799cfpets

7cc07cat

65464son

799cfpets

commit commit

parent

refs/heads/master

.git/HEAD

git://10.0.0.1/git/simpsons.git

blob tree

a134fson

799cfpets

7cc07cat

65464son

799cfpets

commit commit

parent

refs/remotes/

origin/master

refs/heads/master

.git/HEAD

file:///home/sjpark/simpsons

git merge origin/master

blob tree

a134fson

799cfpets

7cc07cat

65464son

799cfpets

commit commit

parent

refs/remotes/

origin/master

refs/heads/

.git/HEAD

blob tree

a134fson

799cfpets

7cc07cat

65464son

799cfpets

commit commit

parent

refs/remotes/

origin/master

refs/heads/

.git/HEAD

Pull Is Just An Abbrev Of Fetch && Merge

May Merge Conflict Occur…

Pull Is Sufficient For Simple Project

Wrap-up

In Short,

Git Is A Content-Addressable File System

Blob, Tree, Commit, Reference. That’s It =3

Thank you :)

http://jeancharpentier.files.wordpress.com/2012/02/capture-plein-c3a9cran-01022012-230955.jpg

Slide-share

http://www.slideshare.net/SeongJaePark1/deep-darkside-ofgit

Latest Version Of This Slide Would Be There

References

http://git-scm.com/bookhttp://www.youtube.com/watch?v=4XpnKHJAok8http://en.wikipedia.org/wiki/The_Simpsons

This work by SeongJae Park is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported

License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/.

Dark Side of Git - We use this on my big data team

Technology