Stretching out for trustworthyreproducible builds
Holger [email protected]
FOSDEM’152015-01-31
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 1 / 108
What are reproducible builds?
“reproducible builds”enable anyone to reproduceidentical binary packages
from a given source
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 3 / 108
“I want to believe”
source binarybuild
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 4 / 108
“I want to believe”
source binarybuild
free softwarefreedomto study
freedomto run
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 5 / 108
“I want to believe”
source binarybuild
can be verified can be used
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 6 / 108
“I want to believe”
source binarybuild
can be verified can be used
prove itto me!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 7 / 108
Why?
Reproducible builds allow for independent verificationsthat a binary matches what the source intended toproduce.
… and other nice things.
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 9 / 108
But I’m the developer!
“I know what’s in the binary because I compiled itmyself!”
“I’m an upstanding, careful, and responsible individual!”
“Why should I have to worry about hypothetical risksabout the contents of my binaries?”
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 10 / 108
But the build machines are secure
How can you be sure?
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 11 / 108
Unpleasant thoughts
We think of software development as a fundamentallybenign activity.
I “I’m not that interesting.”
Users can be targeted through developersKnown successful attacks against infrastructure usedby Linux (2003), FreeBSD (2013)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 12 / 108
Strong motivations
Compromise one computer to get:I Hundreds of millions of other computers?I Every bank account in the world?I Every Windows computer in the world?I Every Linux server in the world?
Compromise one computer is worth:I $100k USD? (Market price of remote 0day)I $100M USD? (Censorship budget of Iran per year)I $4B USD? (Bitcoin market cap)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 13 / 108
How small can a backdoor be?
OpenSSH 3.0.2 (CVE-2002-0083) – exploitable securitybug (privilege escalation: user can get root)
{Channel *c;
- if (id < 0 || id > channels_alloc) {+ if (id < 0 || id >= channels_alloc) {
log("channel_lookup: %d: bad id", id);return;
}
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 14 / 108
Result of fixing the bug (asm)
cmpl $0x0,0x8(%ebp) cmpl $0x0,0x8(%ebp)js 16 js 16mov 0x4,%eax mov 0x4,%eaxcmp %eax,0x8(%ebp) cmp %eax,0x8(%ebp)jle 30 jl 30mov 0x8(%ebp),%eax mov 0x8(%ebp),%eaxmov %eax,0x4(%esp) mov %eax,0x4(%esp)movl $0x4c,(%esp) movl $0x4c,(%esp)call 25 call 25
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 15 / 108
Result of fixing the bug (asm)
cmpl $0x0,0x8(%ebp) cmpl $0x0,0x8(%ebp)js 16 js 16mov 0x4,%eax mov 0x4,%eaxcmp %eax,0x8(%ebp) cmp %eax,0x8(%ebp)jle 30 jl 30mov 0x8(%ebp),%eax mov 0x8(%ebp),%eaxmov %eax,0x4(%esp) mov %eax,0x4(%esp)movl $0x4c,(%esp) movl $0x4c,(%esp)call 25 call 25
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 16 / 108
Resulting difference in the binaryWhat’s the difference between if (a > b)and if (a >= b) in x86 assembly?
assembly: JLE JL
opcode: 0x7E 0x7C
binary: 01111110 01111100
A single bit!
Other corresponding opcode pairs also differ by just asingle bit (JGE=0x7D, JG=0x7F)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 17 / 108
Result of fixing the bug (hex)Vulnerable Fixed55 89 e5 83 ec28 83 7d 08 0078 0a a1 04 0000 00 39 45 087e 1a 8b 45 0889 44 24 04 c704 24 4c 00 0000 e8 fc ff ffff b8 00 00 0000 eb 35
55 89 e5 83 ec28 83 7d 08 0078 0a a1 04 0000 00 39 45 087c 1a 8b 45 0889 44 24 04 c704 24 4c 00 0000 e8 fc ff ffff b8 00 00 0000 eb 35
Overall file size: approx. 500 kBHolger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 18 / 108
Result of fixing the bug (hex)Vulnerable Fixed55 89 e5 83 ec28 83 7d 08 0078 0a a1 04 0000 00 39 45 087e 1a 8b 45 0889 44 24 04 c704 24 4c 00 0000 e8 fc ff ffff b8 00 00 0000 eb 35
55 89 e5 83 ec28 83 7d 08 0078 0a a1 04 0000 00 39 45 087c 1a 8b 45 0889 44 24 04 c704 24 4c 00 0000 e8 fc ff ffff b8 00 00 0000 eb 35
Overall file size: approx. 500 kBHolger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 19 / 108
Bitcoin’s motivation
Malicious modifications to binaries could result inirrevocable unwanted transfers of bitcoinsIndividual developers could be blamed for suchmodificationsUsers might not believe that a developer’s machinewas hackedReproducible builds therefore protect developers
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 21 / 108
Nothing new though
From: Martin Uecker <[email protected]>Cc: [email protected]: Sun, 23 Sep 2007 23:32:59 +0200
I think it would be really cool if the Debian policy requiredthat packages could be rebuild bit-identical from source.At the moment, it is impossible to independly verify theintegricity of binary packages.
https://lists.debian.org/debian-devel/2007/09/msg00746.html
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 22 / 108
Wouldn’t it be cool?
Debian is the largest collection of free softwareMore than 21,000 source packages“Our priorities are our users and free software”
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 23 / 108
How?
Record the build environmentReproduce the build environmentEliminate unneeded variations
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 25 / 108
How:Record the build environment
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 26 / 108
*.buildinfo
New control file *.buildinfo which records:
Versions of build dependenciesI … and their dependencies
Checksum of the source package.Checksums of the binary packages.
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 27 / 108
Example *.buildinfo
Format: 1.9Build-Architecture: amd64Source: txtorconBinary: python-txtorconArchitecture: allVersion: 0.11.0-1Build-Path: /usr/src/debian/txtorcon-0.11.0-1Checksums-Sha256:a26549d9…7b 125910 python-txtorcon_0.11.0-1_all.deb28f6bcbe…69 2039 txtorcon_0.11.0-1.dsc
Build-Environment:base-files (= 8),base-passwd (= 3.5.37),bash (= 4.3-11+b1),…
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 28 / 108
How:Reproduce the build environment
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 29 / 108
snapshot.debian.org
snapshot.debian.org archives every state of theDebian archive.2015-01-29: 28 terabytes of data in almost 17 millionfiles.
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 30 / 108
srebuild
Thin wrapper around sbuildFind the right archive snapshotInstall packages listed in the *.buildinfo fileStart the build
Status: work in progress
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 31 / 108
How:Eliminate unneeded variations
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 32 / 108
General approach
Gitian (Bitcoin, Tor Browser):I Use a VM: same kernel, same user, same build pathI libfaketime
Debian:I Fix the toolsI Fix the build systemsI Work-arounds as last resort
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 33 / 108
strip-nondeterminism
Normalize various file formatsCurrently handle:
I ar archives (.a)I gzipI Java jarI Javadoc HTMLI ZIP archives
Written in Perl (like dpkg-dev)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 34 / 108
Investigating packages
debbindiff :
Two files → HTMLRecursively unpack archivesSeeks human readability:
I uncompress PDFI disassemble binaries,I unpack Gettext files,I … easy to extend to new file formats
Falls back to binary comparison
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 35 / 108
reproducible.debian.net: idea
Build the packageRebuild the packageCompare the results
Initially a 10 lines shell script…
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 37 / 108
reproducible.debian.net: todayDriven by jenkins.debian.net
I announced 2014-09-26
14 jobs defined in 263 LOC of YAMLI 3 build jobs, rescheduled every 2 minutesI Job for auto-schedulingI Jobs to recreate pbuilder base.tgz and debbindiff schrootI Job to update notes and issues from GitI JSON outputI Remaining jobs mostly to create HTML and graphs
13 Shell scripts with 2049 LOC4 Python scripts with 948 LOC
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 38 / 108
reproducible.debian.net: variations
The second build is different in:
timefile orderingCPU ordering and availabilityhostnameuser & grouplocale
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 39 / 108
reproducible.debian.net: more goodies
https://reproducible.debian.net/$PACKAGEReproducibility status in tracker.debian.orgPackage sets:
I Essential:yes, build-essentialI 1337 most frequently installed packages (popcon)I Packages installed on Debian machinesI All packages in Tails and their build dependenciesI All packages in Grml and their build dependencies
List of unreproducible packages per maintainer
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 40 / 108
reproducible.debian.net: not reproducible
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 41 / 108
reproducible.debian.net: package
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 42 / 108
reproducible.debian.net: package
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 43 / 108
reproducible.debian.net: package
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 44 / 108
Experimental “reproducible” toolchain
Test and refine changes to tools used by build systemsCurrently 20 source packagesExamples : dpkg, debhelper, cdbs, libxslt,r-base, gtk-doc, fontforge, …Most patches already submittedgit://git.debian.org/reproducible/$pkg.gitin branch pu/reproducible_builds
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 45 / 108
reproducible.debian.net overall stats
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 46 / 108
For those in the back
81.7%More than 17,600 source packages!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 47 / 108
For those in the back
81.7%More than 17,600 source packages!
(in our test environment!)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 48 / 108
What is not tested (yet)
uname outputDay/month/year/proc/cpuinfoMore?
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 49 / 108
Identified issues
Timestamps (recording current time)File order(Pseudo-)randomness:
I Temporary file pathsI UUIDI Protection against complexity attacks
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 53 / 108
Identified issues (cont.)
CPU and memory related:I Code optimizations for current CPU classI Recording of memory addresses
Build-pathOthers, eg. locale settings
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 54 / 108
Identified issues (cont.)
ExamplesTimestamps added by build systems
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 55 / 108
Timestamps in static libraries
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 56 / 108
Timestamps in static libraries (cont.)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 57 / 108
Timestamps written by Maven
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 58 / 108
Timestamps written by PyQt4
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 59 / 108
Timestamps written by Erlang compiler
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 60 / 108
Timestamps in PE binariesWindows, UEFI, Mono…
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 61 / 108
Timestamps in Java jar
They are ZIP archives, really.
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 62 / 108
Timestamps in PHP registry
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 63 / 108
Timestamps by a template engine
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 64 / 108
Identified issues (cont.)
ExamplesArchives
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 65 / 108
Timestamps in gzip headers
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 66 / 108
Timestamps in ZIP archives
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 67 / 108
Users and groups in tarballs
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 69 / 108
Identified issues (cont.)
ExamplesTimestamps in documentation
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 70 / 108
Timestamps written by Doxygen
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 71 / 108
Timestamps written by docbook-to-man
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 72 / 108
Timestamps written by Ghostscript
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 73 / 108
Timestamps written by Ghostscript (cont.)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 74 / 108
Timestamps written by GNU groff
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 75 / 108
Timestamps written by Javadoc
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 76 / 108
Timestamps written by man2html
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 77 / 108
Timestamps in TeX output (.dvi)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 78 / 108
Identified issues (cont.)
Examples“Compiled at/on/by”
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 79 / 108
Build time via C preprocessor macros
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 80 / 108
Build time via C preprocessor macros
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 81 / 108
Build time recorded via Makefile
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 82 / 108
Hostname recorded via ./configure
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 83 / 108
Build time recorded via ./configure
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 84 / 108
m4 macros for autoconf (build time)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 85 / 108
m4 macros for autoconf (username)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 86 / 108
m4 macros for autoconf (hostname)
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 87 / 108
Identified issues (cont.)
ExamplesFile ordering
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 88 / 108
File ordering in python-support files
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 89 / 108
Identified issues (cont.)
ExamplesRandomness
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 90 / 108
Random Perl hash orderSee Algorithmic complexity attacks in perlsec(1).
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 91 / 108
Random serial numbers in Ogg streams
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 92 / 108
Identified issues (cont.)
ExamplesEven more timestamps!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 93 / 108
Timestamps in PNGEven images!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 94 / 108
Timestamps in TrueType font files
And fonts!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 95 / 108
Please help!
Do not record time, username, hostname, kernelversion…
I … or make it optional.
Sort file paths.Sort dictionary keys.Merge our patches!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 97 / 108
Help Debian?Inventory issuesMake packages build reproduciblyFix known common issues:
I Get reproducible PE binaries;I Random filenames with GCC;I …
Hack Debian archive infrastructureI Store and distribute *.buildinfo files (#763822)I …
Hack tools to display local packages reproducibilitystatus
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 100 / 108
Contributors
akira Hans-Christoph Steiner Peter De WachterAndrew Ayer Helmut Grohne Reiner HerrmannAsheesh Laroia Holger Levsen Stefano RiveraChris Lamb (lamby) josch Stéphane GlonduChris West Lunar Steven ChamberlainDaniel Kahn Gillmor Mattia Rizzolo Tom FitzhenryDavid Suarez Niels Thykier Tomasz BuchertDrew Fisher Niko Tyni WookeyGuillem Jover Paul Wise Ximin Luo
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 101 / 108
Stay in touch
Subscribe to the ReproducibleBuilds wiki pages.Mailing lists:
I [email protected] [email protected]
Join #debian-reproducible on irc.oftc.net
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 102 / 108
Other distributions
Fedorahttp://securityblog.redhat.com/2013/09/18/reproducible-builds-for-fedora/OpenSUSE build-comparehttps://build.opensuse.org/package/show/openSUSE:Factory/build-compareNixOShttp://lists.science.uu.nl/pipermail/nix-dev/2013-June/011357.html
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 103 / 108
Reproducible Fedora?
No known activity after the initial blog post.Fedora is leading developments for key components.Can we help reproducible Fedora?
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 104 / 108
SLES? OpenBSD? Firefox? LibreOffice?We hope reproducible builds will become the norm.Our documentation should be helpful to others.Come talk to us!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 105 / 108
Future
Ready for the next Debian release (stretch)Reproducible installation mediaReproducible live images (pure Debian, Grml, Tails)Reproducible cross-platform builds (“rebootstrap”)Binary transparency logs: public log of all softwarereleases
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 106 / 108
Thanks
Mike Perry and Seth Schoen for their 31C3 talkProfitbricks for sponsoring jenkins.debian.netGlobalsign for sponsoring X.509 certificatesDesigners of Tango iconsEveryone who helped!
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 107 / 108
Questions? Comments?
?https://wiki.debian.org/ReproducibleBuilds
https://reproducible.debian.net/#debian-reproducible on OFTC
Holger B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
Lunar 0603 CCFD 9186 5C17 E88D 4C79 8382 C95C 2902 3DF9
Holger & Lunar (Debian) Stretching reproducible builds FOSDEM’15 108 / 108