@kannonboyPhoto: Le Monde en Vidéo
@kannonboy
@kannonboy@kannonboyPhoto: Le Monde en Vidéo
@kannonboy
Git LFS!
Git LOB!
@kannonboyPhoto: Le Monde en Vidéo
@kannonboy
ok cool
@kannonboyPhoto: Le Monde en Vidéo
Git LFS!
TIM PETTERSEN • SENIOR DEVELOPER • ATLASSIAN • @KANNONBOY
Tracking huge files with Git LFS
@kannonboy
G I T L F S
T H E P R O B L E M W I T H B I G F I L E S
Agenda
C O N V E R T I N G Y O U R R E P O
T I P S F O R T E A M S
@kannonboy
data model
@kannonboy
master
feature/JIRA-123
@kannonboy
434bb..tree
bab1e..parent
Tim P <kannonboy@…> 1455209277 -0800committer
Tim P <kannonboy@…> 1455209277 -0800author
My life is my commit message.
98ca9..
bab1e..
fad3d..git cat-file -p 98ca9$
@kannonboy
git cat-file -p 434bb$
434bb..
98ca9..
bab1e..
fad3d..
ace23..100644 blob .gitignoredbdbd..100644 blob README.mda0bc3..040000 tree app33d33..040000 tree configb1de7..100755 blob deploy-prod.sh7011e..100755 blob deploy-staging.sh
typefilemode SHA-1
@kannonboy
434bb..
98ca9..
bab1e..
fad3d..
ace23.. 1010101
dbdbd..
a0bc3..
33d33..
b1de7..
7011e..
1010101
1010101
1010101
master
1010101
1010101
1010101
@kannonboy
98ca9..
bab1e..
fad3d..
@kannonboy
98ca9..
bab1e..
fad3d..
@kannonboy
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
50mb100mb150mb
@kannonboy
@kannonboy
@kannonboy
(Large File Storage)
Git LFS
@kannonboy
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
$
LFS store
Git host
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
Git host
LFS store
$
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
Git host
LFS store
git push$
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
git pull$
LFS store
Git host
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
git pull$
LFS store
Git host
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
https://git-lfs.github.com/spec/v1version
sha256:325ddfb…oid
29342295size
git cat-file -p 4749d$
4749d..
bdd12..
778aa..
@kannonboy
$ brew install git-lfs
$ git lfs install
@kannonboy
$ cat ~/.gitconfig
[filter "lfs"] clean = git-lfs clean %f smudge = git-lfs smudge %f required = true
@kannonboy
$ git lfs track “*.mp4”
$ cat .gitattributes
*.mp4 filter=lfs diff=lfs merge=lfs -text
@kannonboy
massive_video.mp4
Work tree
dev
.git/lfs/objects
Clean filter(git-lfs clean)
☞
Index
massive_video.mp4
$
.git/objects
git add
@kannonboy
$
dev
.git/lfs/objects
Smudge filter(git-lfs smudge)
Work tree
massive_video.mp4
☞
Commit tree
massive_video.mp4.git/objects
LFS Store
git checkout
@kannonboy
.git/lfs/objects
.git/objectsHosted repo
LFS store
git push / pull
@kannonboy
$ ls .git/hooks/
prepare-commit-msg.sample post-update.sample pre-commit.sample pre-push ...
@kannonboy
$ git push
Git LFS: (12 of 13 files, 1 skipped) 168.75 MB / 180.87 MB, 12.12 skipped
Counting objects: 22, done. ...
@kannonboy
$ git pull
remote: Counting objects: 3, done. ... Downloading massive_video.mp4 (38.79 MB) ... 1 file changed, 2 insertions(+)
@kannonboy
POST .../repo.git
I need object cafebabe
@kannonboy
/info/lfs/objects/batch{“objects”:[
{“oid”: “cafebabe...”,“size”: 40689401}, ...
],“operation”: “download”}
@kannonboy
200 OK
{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {
“download”: { “href”: “https://…/lfs/cafebabe…”,
@kannonboy
cafebabe is over there
} }
...
@kannonboy
200 OK
{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {
“download”: { “href”: “https://…/lfs/cafebabe…”,
@kannonboy
cafebabe is over there
“header”: {“Authorization”: “JWT eyJ0eXA…”,
} } }
...
@kannonboy
Converting to Git LFS
☞
@kannonboy
Converting to Git LFS
☞
@kannonboy
434bb..
fad3d..
98ca9..
41222..
dabad.. 100mb150mb
ace34..☞ 150mb!?!?
@kannonboy
git filter-branch
$ git filter-branch --force --index-filter \ 'git rm --cached --ignore-unmatch big_video.mp4’ \ --prune-empty --tag-name-filter cat -- --all
DON’T DO
THIS!
@kannonboy
$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig
for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all
@kannonboy
DON’T DO
THIS
EITHER!
@kannonboy
BFG Repo-Cleaner
@kannonboy
by @rtyley
@kannonboy
BFG Repo-Cleaner
@kannonboy
10-720x faster than filter-branch
built to kill history
Git LFS support
by @rtyley
@kannonboy
$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig
for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all
@kannonboy
DON’T DO
THIS
EITHER!
@kannonboy
$ brew install bfg
$ bfg —-convert-to-git-lfs ‘*.{zip,mp4}’ --no-blob-protection
@kannonboy
Repofactoring
@kannonboy
@kannonboy
Identifying large objects
github.com/bloomberg/repofactor
by @hashpling
@kannonboy
$
a295ef4… 102437 953722cc7063… 152171 140443
blob SHA size on disk average blob size
generate-larger-than 50000
Identifying large objects
@kannonboy
a295ef4… 102437 95372 2cc7063… 152171 140443
, PNG, 1148 x 482, PNG, 1101 x 800
$ generate-larger-than 50000 \| add-file-info
Identifying large objects
@kannonboy
$ generate-larger-than 50000 \ | add-file-info
2cc7063… 152171 140443, PNG, 1101 x 800 a295ef4… 102437 95372, PNG, 1148 x 482
order by average blob size
\| sort -k3nr
Identifying large objects
@kannonboy
$ generate-larger-than 50000 \ | add-file-info \ | sort -k3nr
$ report-on-large-objects big-stuff.txt
logo-hdpi.png 2cc7063… 152171 140443, PNG…logo-mdpi.png a295ef4… 102437 95372, PNG…
paths
> big-stuff.txt
Identifying large objects
@kannonboy
$ bfg —-convert-to-git-lfs ‘logo-*.png’ --no-blob-protection
Identifying large objects
@kannonboy
Enable in Bitbucket
@kannonboy
@kannonboy
Tips for teams
@kannonboyBeware merge conflicts @kannonboy
@kannonboy
…meanwhile in
@kannonboy
Locking
@kannonboy
Source: etonline.com/music
@kannonboy
master
feature0
feature1
File locking (single branch model)
@kannonboy
master
feature0
feature1
File locking (single branch model)
@kannonboy
master
feature0
feature1
File locking (single branch model)
@kannonboy
master
feature0
feature1
File locking (single branch model)
@kannonboy
master
feature0
feature1
File locking (single branch model)
LAME
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
master
feature0
feature1
File locking (multi branch model)
@kannonboy
$ git lfs lock <path>
$ git lfs unlock <path>
$ git lfs locks [-i id] [-p path]
@kannonboy
Until then…
@kannonboyTeamwork @kannonboyTeamwork
@kannonboy
$ git lfs fetch
master
feature0
feature1
@kannonboy
$ git lfs fetch
$ git config lfs.fetchrecentalways “true”
master
feature0
feature1
--recent
T-minus 7 days
@kannonboy
$ git config lfs.fetchrecentalways “true”
lfs.fetchrecentrefsdays
lfs.fetchrecentremoterefs
lfs.fetchrecentcommitsdays
(default = 7)
(default = 0)
$ git lfs fetch --recent
@kannonboy
$
master
feature0
feature1
git lfs prune
@kannonboy
$
master
feature0
feature1
--recent(7 days)
prune(10 days)
git lfs prune
@kannonboy
lfs.pruneoffsetdays
lfs.pruneverifyremotealways
(default = 3)
Set this!
$ git lfs prune
@kannonboyFetch the bare necessitiesFetch the bare necessities @kannonboy© Disney
@kannonboy
# for a build that just runs the unit tests $ git lfs fetch --exclude Assets/**
# for an audio engineer $ git lfs fetch --include Assets/Audio/**
@kannonboy
# for a build that just runs the unit tests $ git lfs fetch --exclude Assets/**
# for an audio engineer $ git lfs fetch --include Assets/Audio/**
$ git config lfs.fetchexclude Assets/**
$ git config lfs.fetchinclude Assets/Audio/**
@kannonboy
coming very soon!Bitbucket Cloud
git-lfs.github.comdocs
github.com/github/git-lfssource
atlassian.com/bitbucketBitbucket Server
Lookingfor
more?
@kannonboy
coming very soon!Bitbucket Cloud
git-lfs.github.comdocs
github.com/github/git-lfssource
atlassian.com/bitbucketBitbucket Server
Lookingfor
more?
@kannonboy
coming very soon!Bitbucket Cloud
git-lfs.github.comdocs
github.com/github/git-lfssource
atlassian.com/bitbucketBitbucket Server
Lookingfor
more?
Follow me for occasional Git,Bitbucket & JIRA trivia