Date post: | 12-Feb-2017 |
Category: |
Technology |
Upload: | tim-pettersen |
View: | 215 times |
Download: | 2 times |
@kannonboyPhoto: Le Monde en Vidéo
@kannonboy
@kannonboy@kannonboyPhoto: Le Monde en Vidéo
@kannonboy
Git LFS!
Git LOB!
@kannonboyPhoto: Le Monde en Vidéo
@kannonboy
ok cool
@kannonboyPhoto: Le Monde en Vidéo
Git LFS!
TIM PETTERSEN • SENIOR DEVELOPER • ATLASSIAN • @KANNONBOY
Tracking huge files with Git LFS
@kannonboy
G I T L F S
T H E P R O B L E M W I T H B I G F I L E S
T I P S F O R T E A M S
Agenda
C O N V E R T I N G Y O U R R E P O
@kannonboy
data model
@kannonboy
master
feature/JIRA-123
@kannonboy
434bb..tree
bab1e..parent
Tim P <kannonboy@…> 1455209277 -0800committer
Tim P <kannonboy@…> 1455209277 -0800author
My life is my commit message.
98ca9..
bab1e..
fad3d.. git cat-file -p 98ca9$
@kannonboy
git cat-file -p 434bb$
434bb..
98ca9..
bab1e..
fad3d..
ace23..100644 blob .gitignoredbdbd..100644 blob README.mda0bc3..040000 tree app33d33..040000 tree configb1de7..100755 blob deploy-prod.sh7011e..100755 blob deploy-staging.sh
typefilemode SHA-1
@kannonboy
434bb..
98ca9..
bab1e..
fad3d..
ace23.. 1010101
dbdbd..
a0bc3..
33d33..
b1de7..
7011e..
1010101
1010101
1010101
master
1010101
1010101
1010101
@kannonboy
98ca9..
bab1e..
fad3d..
@kannonboy
98ca9..
bab1e..
fad3d..
@kannonboy
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
50mb
100mb
150mb
@kannonboy
@kannonboy
(Large File Storage)
Git LFS
@kannonboy
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
$
LFS store
Git host
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
Git host
LFS store
git push$
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
git pull$
LFS store
Git host
@kannonboy
☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..☞
☞
☞
dabad..
98ca9..
bab1e..
fad3d..
86753..
434bb..
https://git-lfs.github.com/spec/v1version
sha256:325ddfb…oid
29342295size
git cat-file -p 4749d$
4749d..
bdd12..
778aa..
@kannonboy
$ brew install git-lfs
$ git lfs install
@kannonboy
$ cat ~/.gitconfig
[filter "lfs"] clean = git-lfs clean %f smudge = git-lfs smudge %f required = true
@kannonboy
$ git lfs track “*.mp4”
$ cat .gitattributes
*.mp4 filter=lfs diff=lfs merge=lfs -text
@kannonboy
massive_video.mp4
Work tree
dev
.git/lfs/objects
Clean filter(git-lfs clean)
☞
Index
massive_video.mp4
$
.git/objects
git add
@kannonboy
$
dev
.git/lfs/objects
Smudge filter(git-lfs smudge)
Work tree
massive_video.mp4
☞
Commit tree
massive_video.mp4.git/objects
LFS Store
git checkout
@kannonboy
.git/lfs/objects
.git/objectsHosted repo
LFS store
git push / pull
@kannonboy
$ ls .git/hooks/
commit-msg.sample post-update.sample pre-commit.sample pre-push ...
@kannonboy
$ git push
Git LFS: (12 of 13 files, 1 skipped) 168.75 MB / 180.87 MB, 12.12 skipped
Counting objects: 22, done. ...
@kannonboy
$ git pull
remote: Counting objects: 3, done. ... Downloading massive_video.mp4 (38.79 MB) ... 1 file changed, 2 insertions(+)
@kannonboy
$ git clone ssh://git@bitbucke..
Cloning into ‘big_repo’ ... Downloading massive_video.mp4 (38.79 MB) ... Checking out files: 100% (13/13), done.
@kannonboy
Auth
SSH / HTTP
HTTP ONLY
Git server with embedded LFS
Store
@kannonboy
Auth
SSH / HTTP
HTTP ONLY
Git server
Separate LFS store (e.g. S3)
@kannonboy
LFS aware Git server LFS storeDev
git clone https://..
repo pack files
smudge filter
cafebabe is over there
I need object cafebabe
GET …/cafebabe
object cafebabe…
@kannonboy
POST .../repo.git
I need object cafebabe
@kannonboy
/info/lfs/objects/batch{“objects”:[
{ “oid”: “cafebabe...”, “size”: 40689401 }, ...
], “operation”: “download”}
@kannonboy
200 OK
{“objects”:[ {“oid”: “cafebabe…”, “size”: 40689401, “actions”: {
“download”: { “href”: “https://…/lfs/cafebabe…”,
@kannonboy
cafebabe is over there
“header”: { “Authorization”: “JWT eyJ0eXA…”, } } }
...
@kannonboy
LFS aware Git server LFS storeDev
git push
please upload it over there
I want to upload cafebabe
POST …/cafebabe
I uploaded cafebabe
@kannonboy
200 OK
{“objects”:[ ... “actions”: {
“upload”: { “href”: “https://…/lfs/cafebabe…”,
… }
@kannonboy
, “verify”: { “href”: “https://…/lfs/callback”,
… }
} ...
go upload it over there
I uploaded cafebabe
@kannonboy
LFS aware Git server LFS storeDev
git push ssh://…
please upload it over there
I want to upload cafebabe
POST …/cafebabe
I uploaded cafebabe
where is the LFS API?
the LFS API is over there
@kannonboy
$ ssh git@bitbucket git-lfs-authenticate \ project/repo.git download
{ “href”: “https://…/lfs/objects/batch”, “header”: { “Authorization”: “JWT eyJ0eXA...” } }
where is the LFS API?
the LFS API is over there
@kannonboy
@kannonboy
LFS aware Git server LFS storeDev
git clone https://..
repo data
POST /info/lfs/objects/batch
LFS objects hypermedia
GET …/<objectSHA>
smudge filter
happens once per
file checked out
@kannonboy
LFS storeDev
git lfs clone https://..
repo data
GET …/<objectSHA>
batched
smudge filter
subtle difference!
POST /info/lfs/objects/batch
LFS objects hypermedia
LFS aware Git server
@kannonboy
Converting to Git LFS
☞
@kannonboy
434bb..
fad3d..
98ca9..
41222..
dabad..100mb
150mbace34..
☞ 150mb!?!?
@kannonboy
git filter-branch
$ git filter-branch --force --index-filter \ 'git rm --cached --ignore-unmatch big_video.mp4’ \ --prune-empty --tag-name-filter cat -- --all
DON’T DO
THIS!
@kannonboy
$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig
for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all
@kannonboy
DON’T DO
THIS
EITHER!
@kannonboy
BFG Repo-Cleaner
@kannonboy
by @rtyley
@kannonboy
BFG Repo-Cleaner
@kannonboy
10-720x faster than filter-branch
built to kill history
Git LFS support
by @rtyley
@kannonboy
$ git filter—branch --prune-empty --tree-filter ' git config -f .gitconfig lfs.url “https://bitbucket.example.com/team/repo.git” git lfs track "*.mp4" git add .gitattributes .gitconfig
for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do git rm -f --cached ${file} git add ${file} done' --tag-name-filter cat -- --all
@kannonboy
DON’T DO
THIS
EITHER!
@kannonboy
$ brew install bfg
$ bfg —-convert-to-git-lfs ‘*.{zip,mp4}’ --no-blob-protection
@kannonboy
Repofactoring
@kannonboy
@kannonboy
Identifying large objects
github.com/bloomberg/repofactor
by @hashpling
@kannonboy
$
a295ef4… 102437 95372 2cc7063… 152171 140443
blob SHA size on disk average blob size
generate-larger-than 50000
Identifying large objects
@kannonboy
a295ef4… 102437 95372 2cc7063… 152171 140443 , PNG, 1148 x 482 , PNG, 1101 x 800
$ generate-larger-than 50000 \ | add-file-info
Identifying large objects
@kannonboy
$ generate-larger-than 50000 \ | add-file-info
2cc7063… 152171 140443, PNG, 1101 x 800 a295ef4… 102437 95372, PNG, 1148 x 482
order by average blob size
\ | sort -k3nr
Identifying large objects
@kannonboy
$ generate-larger-than 50000 \ | add-file-info \ | sort -k3nr
$ report-on-large-objects big-stuff.txt
logo-hdpi.png 2cc7063… 152171 140443, PNG… logo-mdpi.png a295ef4… 102437 95372, PNG…
paths
> big-stuff.txt
Identifying large objects
@kannonboy
$ bfg —-convert-to-git-lfs ‘logo-*.png’ --no-blob-protection
Identifying large objects
@kannonboy
Enable in Bitbucket
@kannonboy
@kannonboy
Tips for teams
@kannonboyBeware merge conflicts @kannonboy
@kannonboy
…meanwhile in
@kannonboy
…meanwhile in
@kannonboyTeamwork @kannonboyTeamwork
@kannonboy
$ git lfs fetch
$ git config lfs.fetchrecentalways “true”
master
feature0
feature1
--recent
T-minus 7 days
@kannonboy
$ git config lfs.fetchrecentalways “true”
lfs.fetchrecentrefsdays
lfs.fetchrecentremoterefs
lfs.fetchrecentcommitsdays
(default = 7)
(default = 0)
$ git lfs fetch --recent
@kannonboy
$ git lfs prune
lfs.pruneoffsetdays
lfs.pruneverifyremotealways
(default = 3)
@kannonboyFetch the bare necessitiesFetch the bare necessities @kannonboy© Disney
@kannonboy
# for a build that just runs the unit tests $ git lfs fetch --exclude Assets/**
# for an audio engineer $ git lfs fetch --include Assets/Audio/**
$ git config lfs.fetchexclude Assets/**
$ git config lfs.fetchinclude Assets/Audio/**
@kannonboyIDEs & GUI tools @kannonboy
@kannonboy
@kannonboy
1. Install EGit Team Provider version 4.2+ 2. Make sure git-lfs is on your path
@kannonboy
@kannonboy
@kannonboy
SourceTree
@kannonboy
coming soon!
Bitbucket Cloud
git-lfs.github.comdocs
github.com/github/git-lfs
source
atlassian.com/bitbucket
Bitbucket Server
Lookingfor
more?
Follow me for occasional Git, Bitbucket & JIRA trivia