Python Packaging GuideRelease

PyPA

January 23, 2014

Contents

1 Introduction 31.1 What’s covered in the guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Where to get packaging support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 How to help packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.4 About the guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5 About ‘Page Status’ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Installation & Packaging Quickstart 52.1 Install the Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Create a Virtual Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.3 Install Python Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.4 Cache Wheels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 Create your own Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.6 Build & Upload your Project to PyPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Tool Recommendations 93.1 Installation Tool Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Packaging Tool Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 The Future of Python Packaging 114.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.2 Presentations & Articles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.3 Completed work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.4 Work in Progress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.5 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5 Advanced Topics 155.1 Technical Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.2 Platform Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.3 Binary Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6 Project Summaries 256.1 pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.2 virtualenv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.3 setuptools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.4 wheel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.5 distlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.6 Warehouse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.7 buildout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

i

6.8 bento . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.9 conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266.10 Hashdist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

7 PEP Summaries 297.1 PEP376 Database of Installed Python Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 297.2 PEP425 Compatibility Tags for Built Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 297.3 PEP427 The Wheel Binary Package Format 1.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307.4 PEP438 Transitioning to release-file hosting on PyPI . . . . . . . . . . . . . . . . . . . . . . . . . . 307.5 PEP453 Explicit bootstrapping of pip in Python installations . . . . . . . . . . . . . . . . . . . . . . 307.6 PEP426 Metadata for Python Software Packages 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . 307.7 PEP440 Version Identification and Dependency Specification . . . . . . . . . . . . . . . . . . . . . 317.8 PEP458 Surviving a Compromise of PyPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

8 A Packaging Timeline 33

9 Glossary 35

ii

Python Packaging Guide, Release

Page Status Incomplete 1

Last Reviewed 2014-01-22

The “Python Packaging User Guide” (PUG) aims to be the authoritative resource on how to package and installdistributions in Python using current tools, but also on the efforts to improve Python packaging.

The guide is part of a larger effort to improve all of the packaging and installation docs, including pip, setuptools,virtualenv, wheel, and docs.python.org. Ultimately, users need more than a guide to feel confident about the currenttools. They need complete, accurate and inter-consistent documentation across all the projects.

The guide is maintained here by the Python Packaging Authority (PyPA). It was forked from the “Hitchhikers Guideto Packaging” in March 2013.

1 The main index page will only be marked “Complete”, when all pages are marked “Complete”, which they are not.

Contents 1

Python Packaging Guide, Release

2 Contents

CHAPTER 1

Introduction

Page Status Complete

Last Reviewed 2014-01-22

1.1 What’s covered in the guide

• An Installation & Packaging Quickstart for getting going fast.

• Recommendations & Justifications for what tools you should be using now.

• The Future of Python Packaging.

• Advanced Topics in Packaging and Installation.

• Summaries of the most relevant packaging projects.

• Summaries of the most relevant packaging PEPs.

• A Packaging Timeline.

• A Glossary of packaging and installation terms.

1.2 Where to get packaging support

• For support related to a specific project, see the mailing list and IRC links on the Projects page.

• For something more general, or when you just not sure, use the distutils-sig list.

1.3 How to help packaging

• Get involved with one of mainstream packaging projects.

• Help us catalog and discuss the current problems in packaging and installation. See the The issue tracker for theproblems in packaging maintained by PyPA.

• Discuss PEPs and long terms plans at distutils-sig.

3

Python Packaging Guide, Release

1.4 About the guide

The guide is maintained here on Github by the Python Packaging Authority (PyPA). It was forked from the “HitchhikersGuide to Packaging” in March 2013. Please Log Issues and Pull Requests to help make the guide better.

1.5 About ‘Page Status’

Each page, even this one, will state at the top whether it’s “Complete” or “Incomplete”. Admittedly, a binary distinctionis not very precise, but most pages will additionally offer a footnote to better explain what’s not complete or wrong.

Also, each page will give a “Last Reviewed” date, wich will only be updated by a PyPA member when a meaningfulreview is done to determine the status.

Why do this? See here.

4 Chapter 1. Introduction

CHAPTER 2

Installation & Packaging Quickstart

Page Status InComplete

Last Reviewed 2014-01-22

2.1 Install the Tools

1. Securely Download get-pip.py 1

2. Run python get-pip.py (this will install pip and setuptools) 2 3

3. Run pip install virtualenv

2.2 Create a Virtual Environment

virtualenv myVEsource myVE/bin/activate

Or, if you want to install packages globally, don’t do this.

2.3 Install Python Packages

Install SomePackage and it’s dependencies from PyPI using Requirement Specifiers

pip install SomePackage # latest versionpip install SomePackage==1.0.4 # specific versionpip install ’SomePackage>=1.0.4’ # minimum version

Install a list of requirements specified in a Requirements File.

pip install -r requirements.txt

Upgrade an already installed SomePackage to the latest from PyPI.

1 “Secure” in this context means using a modern browser or a tool like curl that verifies SSL certificates when downloading from https URLs.2 Depending on your platform, this may require root or Administrator access.3 On Linux and OSX, these tools will usually be available for the system python from a system package manager (e.g. yum or apt-get for linux,

or homebrew for OSX). Unfortunately, there is often delay in getting the latest version this way, so in most cases, you’ll want to use the instructions.

5

Python Packaging Guide, Release

pip install --upgrade SomePackage

Install a project from VCS in “editable” mode. For a full breakdown of the syntax, see pip’s section on VCS Support.

pip install -e git+https://git.repo/some_pkg.git#egg=SomePackage # from gitpip install -e hg+https://hg.repo/some_pkg.git#egg=SomePackage # from mercurialpip install -e svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage # from svnpip install -e git+https://git.repo/some_pkg.git@feature#egg=SomePackage # from a branch

Install a particular source archive file.

pip install ./downloads/SomePackage-1.0.4.tar.gzpip install http://my.package.repo/SomePackage-1.0.4.zip

Install from an alternate index

pip install --index-url http://my.package.repo/simple/ SomePackage

Search an additional index during install, in addition to PyPI

pip install --extra-index-url http://my.package.repo/simple SomePackage

Install from a local directory containing archives (and don’t check PyPI)

pip install --no-index --find-links=file:///local/dir/ SomePackagepip install --no-index --find-links=/local/dir/ SomePackagepip install --no-index --find-links=relative/dir/ SomePackage

Find pre-release and development versions, in addition to stable versions. By default, pip only finds stable versions.

pip install --pre SomePackage

For more on installation, see the pip docs.

2.4 Cache Wheels

FIXME, cover ’pip wheel’

2.5 Create your own Project

See the PyPA sample project.

You can can copy and edit from that to get your project going.

To install your project in “develop” or “editable” mode (i.e. to have your project installed, but still editable fordevelopment)

cd myprojectpython setup.py develop # the setuptools waypip install -e . # the pip way

For more information on creating projects, see:

• Setuptools Developer Guide

• Open Sourcing a Python Project the Right Way

6 Chapter 2. Installation & Packaging Quickstart

Python Packaging Guide, Release

2.6 Build & Upload your Project to PyPI

Build a source distribution

python setup.py sdist

Build a wheel (for advice on when, see Should you upload wheels to PyPI?)

pip install wheelpython setup.py bdist_wheel

Upload your distributions with twine

pip install twinetwine upload dist/*

2.6. Build & Upload your Project to PyPI 7

Python Packaging Guide, Release

8 Chapter 2. Installation & Packaging Quickstart

CHAPTER 3

Tool Recommendations

Page Status Incomplete

Last Reviewed 2014-01-22

If you’re familiar with Python packaging and installation, and just want to know what tools are currently recommended,then here it is.

If you’re interested in learning more about current areas of development, see The Future of Python Packaging.

3.1 Installation Tool Recommendations

• Use pip to install Python packages from PyPI. 1 2

• Use virtualenv to isolate application specific dependencies from a shared Python installation. 3

• Use pip wheel to create a cache of wheel distributions, for the purpose of speeding up subsequent installations.4

• If you’re looking for management of fully integrated cross-platform software stacks, consider buildout (primar-ily focused on the web development community) or Hashdist, or conda (both primarily focused on the scientificcommunity).

FIXME

1) Why virtualenv (what about pyenv? buildout?)2) Why pick *one* installer to recommend?

the rationale from PEP4533) why pip?

- feature breakdown in the "pip vs easy_install" chart- Decoupled from packaging tools.- what easy_install bugs mentioned in PEP453 does pip address?

1 If you need to install from the Egg format (which pip doesn’t support), you can use easy_install (from setuptools) or buildout. Eggs areintended to be replaced by Wheels, so they should become less common over time.

2 The acceptance of PEP453 means that pip will likely be available by default in most installations of Python 3.4 or later.3 The acceptance of PEP453 means that users of Python 3.4 or later will likely be able to use the standard library’s own pyvenv tool instead of

virtualenv. However, using virtualenv will still be recommended for users that need cross-version consistency.4 For more information, see the pip guide to Building and Installing Wheels.

9

Python Packaging Guide, Release

3.2 Packaging Tool Recommendations

• Use setuptools to define projects and build distributions. 5 6

• If your project includes binary extensions, use the bdist_wheel setuptools extension available from the wheelproject, to create and publish wheel distributions for Windows and Mac OS X for your project on PyPI. Thesewheel files should be compatible with the binary installers provided for download from python.org. 7

• Use twine for uploading distributions to PyPI.

FIXME

1. Why setuptools (and not distutils)?2. mention that distutils2 is not a live project anymore3. twine? what about ’setup.py upload’

5 distribute (a fork of setuptools) was merged back into setuptools in June 2013, thereby making setuptools the default choice for packaging.6 When building from source (rather than installing from a wheel file), pip ensures that packages that use the standard library’s distutils

module are built with setuptools instead.7 pip and the wheel format don’t currently offer good tools for handling arbitrary external binary dependencies. Accordingly, PyPI currently only

allows platform specific wheel distributions to be uploaded for Windows and Mac OS X. External binary dependencies are currently best handled bybuilding custom wheel files with the correct dependencies, by using one of the fully integrated cross-platform software stack management systemsmentioned in the installation tools section, or by using platform specific tools.

10 Chapter 3. Tool Recommendations

CHAPTER 4

The Future of Python Packaging

Page Status Complete

Last Reviewed 2014-01-19

The distutils cross-platform build and distribution system was added to the Python standard library in late 1998. Thismeans the current Python software distribution ecosystem (which builds on distutils) has a foundation that is almost15 years old, which poses a variety of challenges to successful evolution.

The current effort to improve the situation started when the packaging library (also known as distutils2)failed to be accepted into the standard library for Python 3.3. That effort had spanned from 2009 to 2012. Whilepackaging itself is no longer under development, it did help lay the foundations for the current effort.

The current effort is managed by the Python Packaging Authority (PyPA), in cooperation with members of the Pythoncore development team.

4.1 Goals

• To provide a relatively easy to use software distribution infrastructure that is also fast, reliable and reasonablysecure. “Reasonably secure”, due to backwards compatibility constraints preventing turning off some insecurelegacy features.

• To improve the docs for users, including the Python Packaging User Guide, anything related to packaging ondocs.python.org, and the project docs for pip, setuptools, virtualenv, and wheel.

• To be progressive, but also be very mindful to not break things that are currently working, due to haste.

• To specifically not focus at first on adding something to the standard library as our solution to our packagingproblems. Adding something to the standard library is hard, and once it’s added, it’s a slow process to changeit. Most of the current effort is largely focused on 3rd party projects.

4.2 Presentations & Articles

In addition to this document, there have been some talks and presentations regarding current and future efforts relatedto packaging.

• PyCon US, March 2013

– Directions in Packaging Q & A Panel (aka ”./setup.py install must die”)

• PyCon AU, July 2013

11

Python Packaging Guide, Release

– Nobody Expects the Python Packaging Authority 1

• Personal essays

– Nick Coghlan

4.3 Completed work

4.3.1 2013

• distlib started releasing to PyPI, and pip began depending on it

• Core PyPI infrastructure relocated to OSU/OSL (with significantly increased resources)

• The core packaging projects were collected under the Python Packaging Authority (PyPA) accounts on GitHuband Bitbucket

• distribute merged back into setuptools, and setuptools development migrated to the PyPA BitBucket account

• PyPI supports clients using verified SSL with standard cert bundles

• PyPI forces web users over to SSL

• pip (>=1.3) and easy_install (>=0.7) use verified SSL by default

• easy_install supports additional hashes beyond md5 (pip already did)

• Fastly CDN enabled for PyPI (donated)

• Restructured the pip install docs to clarify that setuptools and pip are the “base” of the bootstrapping hierarchy

• setuptools available as a cross platform wheel on PyPI

• PEP438 Transitioning to release-file hosting on PyPI and the associated pip changes.

4.3.2 2014

• virtualenv installs pip & setuptools using wheels.

• pip (>=1.5.1) available as a cross platform wheel on PyPI

• pip (>=1.5.1) doesn’t require setuptools to install wheels

• get-pip.py installs setuptools for you, if you don’t already have it

4.4 Work in Progress

• PEP453: Having pip be available by default in Python 3.4 distributions

• PEP449: Removal of the DNS based mirror autodiscovery

1 Repeated that at PyTexas, but can’t find a video link for it. Anyone?

12 Chapter 4. The Future of Python Packaging

Python Packaging Guide, Release

4.5 Future Work

4.5.1 2014

• Migration from the legacy PyPI server to Warehouse (the preview is available at https://preview-pypi.python.org/running off the live PyPI data)

• “Metadata 2.0” (PEP426 / PEP440)

• pip should get a real dependency resolver

• Removal of older pip commands and options that aren’t popular or well maintained (#906, #1046)

• Public release of the Python Packaging User Guide

• Improved handling of in-place pip upgrades on Windows

4.5.2 TBD

• PEPs for “sdist 2.0” and wheel 1.1

• A “MetaBuild” PEP that would allow projects to specify alternative build systems (i.e. something other thansetuptools).

• PEP458: An integration of PyPI with the “The Update Framework (TUF)”

• An update of PEP376 (the installation format) to be json based

• An internal stable api for pip

• Improved PyPI upload API

• Improved local PyPI hosting solutions (e.g. like devpi)

4.5. Future Work 13

Python Packaging Guide, Release

14 Chapter 4. The Future of Python Packaging

CHAPTER 5

Advanced Topics

Page Status Incomplete

Last Reviewed 2013-12-08

This section covers a variety of packaging concepts and topics that don’t fit neatly into the documentation of any par-ticular tool, and don’t really belong in the main guides either. While the entries are organised into different categories,those categorisations are quite broad and flexible.

5.1 Technical Details

Page Status Incomplete

Last Reviewed 2014-01-22

This section covers miscellaneous technical details of pip and other components of the distribution toolchain.

5.1.1 Packaging Formats

FIXME

1) sdist and wheel are the most relevant currently2) what defines an sdist? (sdist 2.0 is coming)

5.1.2 Installation Schemes

FIXME

1. distutils/sysconfig schemes2. global vs user installs3. virtual environments

5.1.3 pip vs easy_install

easy_install was released in 2004, as part of setuptools. It was notable at the time for installing distributions fromPyPI using requirement specifiers, and automatically installing dependencies.

15

Python Packaging Guide, Release

pip came later in 2008, as alternative to easy_install, although still largely built on top of setuptools components. Itwas notable at the time for not installing packages as Eggs or from Eggs (but rather simply as ‘flat’ packages fromsdists), and introducing the idea of Requirements Files, which gave users the power to easily replicate environments.

Here’s a breakdown of the important differences between pip and easy_install now:

pip easy_installInstalls fromWheels

Yes No

UninstallPackages

Yes (pip uninstall) No

DependencyOverrides

Yes (Requirements Files) No

List InstalledPackages

Yes (pip list and pipfreeze)

No

PEP438 Support Yes NoInstallationformat

‘Flat’ packages with egg-infometadata.

Encapsulated Egg format

sys.pathmodification

No Yes

Installs fromEggs

No Yes

pylaunchersupport

No Yes 1

DependencyResolution

Kinda Kinda

Multi-versionInstalls

No Yes

5.1.4 easy_install and sys.path

FIXME

- global easy_install’d packages override --user installs

5.1.5 Wheel vs Egg

FIXME

5.1.6 Multi-version Installs

easy_install allows simultaneous installation of different versions of the same package into a single environmentshared by multiple programs which must require the appropriate version of the package at run time (usingpkg_resources).

For many use cases, virtual environments address this need without the complication of the require directive.However, the advantage of parallel installations within the same environment is that it works for an environmentshared by multiple applications, such as the system Python in a Linux distribution.

The major limitation of pkg_resources based parallel installation is that as soon as you import pkg_resourcesit locks in the default version of everything which is already available on sys.path. This can cause problems, since

1http://pythonhosted.org/setuptools/easy_install.html#natural-script-launcher

16 Chapter 5. Advanced Topics

Python Packaging Guide, Release

setuptools created command line scripts use pkg_resources to find the entry point to execute. This meansthat, for example, you can’t use require tests invoked through nose or a WSGI application invoked throughgunicorn if your application needs a non-default version of anything that is available on the standard sys.path- the script wrapper for the main application will lock in the version that is available by default, so the subsequentrequire call in your own code fails with a spurious version conflict.

This can be worked around by setting all dependencies in __main__.__requires__ before importingpkg_resources for the first time, but that approach does mean that standard command line invocations of theaffected tools can’t be used - it’s necessary to write a custom wrapper script or use python -c ’<commmand>’to invoke the application’s main entry point directly.

Refer to the pkg_resources documentation for more details.

5.1.7 Dependency Resolution

FIXME

what to cover:- pip lacking a true resolver (currently, "1st found wins"; practical for overriding in requirements files)- easy_install will raise an error if mutually-incompatible versions of a dependency tree are installed.- console_scripts complaining about conflicts- scenarios to breakdown:

- conficting dependencies within the dep tree of one argument ‘‘pip|easy_install OnePackage‘‘- conflicts across arguments: ‘‘pip|easy_install OnePackage TwoPackage‘‘- conflicts with what’s already installed

5.2 Platform Integration

Page Status Incomplete

Last Reviewed 2014-01-22

This section covers various platform specific integration issues and usage models, along with a discussion of anyparticularly problematic areas Python users might expect to encounter.

5.2.1 Installing on Debian/Ubuntu

FIXME

- dist-packages vs site-pacakges- /usr vs /usr/local- users getting stuck on older setuptools use LTS distros

5.2.2 Installing on CentOS/RedHat

FIXME

5.2.3 Installing on Windows

FIXME

5.2. Platform Integration 17

Python Packaging Guide, Release

5.2.4 Installing on OSX

FIXME

- inability to override default setuptools with --user install(https://github.com/pypa/pip/issues/1480)

5.2.5 Building RPMs for Python projects

FIXME

5.2.6 Building debs for Python projects

FIXME

5.2.7 Building Windows installers for Python projects

FIXME

5.2.8 Building Mac OS X installers for Python projects

FIXME

5.2.9 Installing Scientific Packages

Scientific software tends to have more complex dependencies than most, and it will often have multiple build optionsto take advantage of different kinds of hardware, or to interoperate with different pieces of external software.

In particular, NumPy, which provides the basis for most of the software in the scientific Python stack can be configuredto interoperate with different FORTRAN libraries, and can take advantage of different levels of vectorised instructionsavailable in modern CPUs.

Unfortunately, as of December 2013, given NumPy’s current build and distribution model, the standard tools currentlyaren’t quite up to the task of distributing pre-built NumPy binaries, as most users aren’t going to know which versionthey need, and the wheel format currently doesn’t allow the installer to make that decision on the user’s behalf atinstall time.

It is expected that this situation will eventually be resolved either by future iterations of the standard tools providingfull support for the intricacies of NumPy’s current build and distribution process, or by the NumPy developers choosingone build variant as the “lowest acceptable common denominator” and publishing that as a wheel file on PyPI.

In the meantime, however, there are a number of alternative options for obtaining scientific Python libraries (or anyother Python libraries that require a compilation environment to install from source and don’t provide pre-built wheelfiles on PyPI).

Building from source

The same complexity which makes it difficult to distribute NumPy (and many of the projects that depend on it) aswheel files also make them difficult to build from source yourself. However, for intrepid folks that are willing to spendthe time wrangling compilers and linkers for both C and FORTRAN, building from source is always an option.

18 Chapter 5. Advanced Topics

Python Packaging Guide, Release

Linux distribution packages

For Linux users, the system package manager will often have pre-compiled versions of various pieces of scientificsoftware, including NumPy and other parts of the scientific Python stack.

If using versions which may be several months old is acceptable, then this is likely to be a good option (just make sureto allow access to packages installed into the system Python when using virtual environments).

Windows installers

Many Python projects that don’t (or can’t) currently publish wheel files at least publish Windows installers, eitheron PyPI or on their project download page. Using these installers allows users to avoid the need to set up a suitableenvironment to build extensions locally.

The extensions provided in these installers are typically compatible with the CPython Windows installers publishedon python.org.

For projects which don’t provide their own Windows installers (and even some which do), Christoph Gohlke at theUniversity of California provides a collection of Windows installers. Many Python users on Windows have reported apositive experience with these prebuilt versions.

As with Linux system packages, the Windows installers will only install into a system Python installation - they donot support installation in virtual environments. Allowing access to packages installed into the system Python whenusing virtual environments is a common approach to working around this limitation.

The wheel project also provides a wheel convert subcommand that can convert a Windows bdist_wininst installer to awheel.

Mac OS X installers and package managers

Similar to the situation on Windows, many projects (including NumPy) publish Mac OS X installers that are compati-ble with the Mac OS X CPython binaries published on python.org.

Mac OS X users also have access to Linux distribution style package managers such as MacPorts. The SciPy sitehas more details on using MacPorts to install the scientific Python stack

SciPy distributions

The SciPy site lists several distributions that provide the full SciPy stack to end users in an easy to use and updateformat.

Some of these distributions may not be compatible with the standard pip and virtualenv based toolchain.

The conda cross-platform package manager

Anaconda (in this context) is a SciPy distribution published by Continuum Analytics.

conda is the open source (BSD licensed) package management system that Continuum Analytics created and pub-lished as part of Anaconda’s development. It doesn’t support interoperability with system package managers the waythe standard toolchain does (as, unlike the standard toolchain, that isn’t one of conda’s design goals), but it does sup-port some degree of interoperability with the standard toolchain itself. In particular, bootstrapping conda via pipinstall conda and then running the conda init command provides access to all of the pre-built binaries thatContinuum Analytics have created for the free version of the Anaconda distribution.

5.2. Platform Integration 19

Python Packaging Guide, Release

At time of writing (December, 2013), there are still some rough edges when attempting to use conda to install Ana-conda packages outside the Anaconda distribution, but those issues are still likely to be simpler to resolve or workaround than building local versions of scientific Python libraries and their external dependencies.

5.3 Binary Extensions

Page Status Incomplete

Last Reviewed 2013-12-08

One of the features of the CPython reference interpreter is that, in addition to allowing the execution of Python code, italso exposes a rich C API for use by other software. One of the most common uses of this C API is to create importableC extensions that allow things which aren’t always easy to achieve in pure Python code.

5.3.1 An overview of binary extensions

Use cases

The typical use cases for binary extensions break down into just three conventional categories:

• accelerator modules: these modules are completely self-contained, and are created solely to run faster than theequivalent pure Python code runs in CPython. Ideally, accelerator modules will always have a pure Pythonequivalent to use as a fallback if the accelerated version isn’t available on a given system. The CPython standardlibrary makes extensive use of accelerator modules.

• wrapper modules: these modules are created to expose existing C interfaces to Python code. They may eitherexpose the underlying C interface directly, or else expose a more “Pythonic” API that makes use of Python lan-guage features to make the API easier to use. The CPython standard library makes extensive use of acceleratormodules.

• low level system access: these modules are created to access lower level features of the CPython runtime, theoperating system, or the underlying hardware. Through platform specific code, extension modules may achievethings that aren’t possible in pure Python code. A number of CPython standard library modules are written in Cin order to access interpreter internals that aren’t exposed at the language level.

One particularly notable feature of C extensions is that, when they don’t need to call back into the interpreter run-time, they can release CPython’s global interpreter lock around long-running operations (regardling of whetherthose operations are CPU or IO bound).

Not all extension modules will fit neatly into the above categories. The extension modules included with NumPy, forexample, span all three use cases - they moves inner loops to C for speed reasons, wrap external libraries written in C,FORTRAN and other languages, and uses low level system interfaces for both CPython and the underlying operationsystem to support concurrent execution of vectorised operations and to tightly control the exact memory layout ofcreated objects.

Disadvantages

The main disadvantage of using binary extensions is the fact that it makes subsequent distribution of the software moredifficult. One of the advantages of using Python is that it is largely cross platform, and the languages used to writeextension modules (typically C or C++, but really any language that can bind to the CPython C API) typically requirethat custom binaries be created for different platforms.

This means that binary extensions:

20 Chapter 5. Advanced Topics

Python Packaging Guide, Release

• require that end users be able to either build them from source, or else that someone publish pre-built binariesfor common platforms

• may not be compatible with different builds of the CPython reference interpreter

• often will not work correctly with alternative interpreters such as PyPy, IronPython or Jython

• if handcoded, make maintenance more difficult by requiring that maintainers be familiar not only with Python,but also with the language used to create the binary extension, as well as with the details of the CPython C API.

• if a pure Python fallback implementation is provided, make maintenance more difficult by requiring that changesbe implemented in two places, and introducing additional complexity in the test suite to ensure both versionsare always executed.

Another disadvantage of relying on binary extensions is that alternative import mechanisms (such as the ability toimport modules directly from zipfiles) often won’t work for extension modules (as the dynamic loading mechanismson most platforms can only load libraries from disk).

Alternatives to handcoded accelerator modules

When extension modules are just being used to make code run faster (after profiling has identified the code where thespeed increase is worth additional maintenance effort), a number of other alternatives should also be considered:

• look for existing optimised alternatives. The CPython standard libary includes a number of optimised datastructures and algorithms (especially in the builtins and the collections and itertools modules). ThePython Package Index also offers additional alternatives. Sometimes, the appropriate choice of standard libraryor third party module can avoid the need to create your own accelerator module.

• for long running applications, the JIT compiled PyPy interpreter may offer a suitable alternative to the standardCPython runtime. The main barrier to adopting PyPy is typically reliance on other binary extension modules- while PyPy does emulate the CPython C API, modules that rely on that cause problems for the PyPy JIT,and the emulation layer can often expose latent defects in extension modules that CPython currently tolerates(frequently around reference counting errors - an object having one live reference instead of two often won’tbreak anything, but no references instead of one is a major problem).

• Cython is a mature static compiler that can compile most Python code to C extension modules. The initialcompilation provides some speed increases (by bypassing the CPython interpreter layer), and Cython’s optionalstatic typing features can offer additional opportunities for speed increases. Using Cython still has the disad-vantage of increasing the complexity of distributing the resulting application, but has the benefit of having areduced barrier to entry for Python programmers (relative to other languages like C or C++).

• Numba is a newer tool, created by members of the scientific Python community, that aims to leverage LLVMto allow selective compilation of pieces of a Python application to native machine code at runtime. It requiresthat LLVM be available on the system where the code is running, but can provide significant speed increases,especially for operations that are amenable to vectorisation.

Alternatives to handcoded wrapper modules

The C ABI (Application Binary Interface) is a common standard for sharing functionality between multiple applica-tions. One of the strengths of the CPython C API (Application Programming Interface) is allowing Python users totap into that functionality. However, wrapping modules by hand is quite tedious, so a number of other alternativeapproaches should be considered.

The approaches described below don’t simplify the distribution case at all, but they can significantly reduce the main-tenance burden of keeping wrapper modules up to date.

5.3. Binary Extensions 21

Python Packaging Guide, Release

• In addition to being useful for the creation of accelerator modules, Cython is also useful for creating wrappermodules. It still involves wrapping the interfaces by hand, however, so may not be a good choice for wrappinglarge APIs.

• cffi is a project created by some of the PyPy developers to make it straightforward for developers that alreadyknow both Python and C to expose their C modules to Python applications. It also makes it relatively straight-forward to wrap a C module based on its header files, even if you don’t know C yourself.

One of the key advantages of cffi is that it is compatible with the PyPy JIT, allowing CFFI wrapper modulesto participate fully in PyPy’s tracing JIT optimisations.

• SWIG is a wrapper interface generator that allows a variety of programming languages, including Python, tointerface with C and C++ code.

• The standard library’s ctypes module, while useful for getting access to C level interfaces when header infor-mation isn’t available, suffers from the fact that it operates solely at the C ABI level, and thus has no automaticconsistency checking between the interface actually being exported by the library and the one declared in thePython code. By contrast, the above alternatives are all able to operate at the C API level, using C header filesto ensure consistency between the interface exported by the library being wrapped and the one expected by thePython wrapper module. While cffi can operate directly at the C ABI level, it suffers from the same interfaceinconsistency problems as ctypes when it is used that way.

Alternatives for low level system access

For applications that need low level system access (regardless of the reason), a binary extension module often isthe best way to go about it. This is particularly true for low level access to the CPython runtime itself, since someoperations (like releasing the Global Interpreter Lock) are simply invalid when the interpreter is running code, even ifa module like ctypes or cffi is used to obtain access to the relevant C API interfaces.

For cases where the extension module is manipulating the underlying operating system or hardware (rather than theCPython runtime), it may sometimes be better to just write an ordinary C library (or a library in another systemsprogramming language like C++ or Rust that can export a C compatible ABI), and then use one of the wrappingtechniques described above to make the interface available as an importable Python module.

5.3.2 Implementing binary extensions

mention the stable ABI (3.2+, link to the CPython C API docs)mention the module lifecyclemention the challenges of shared static state and subinterpretersmention the implications of the GIL for extension modulesmention the memory allocation APIs in 3.4+

mention again that all this is one of the reasons why you probably

*don’t* want to handcode your extension modules :)

5.3.3 Building binary extensions

FIXME

cover Windows binary compatibility requirementscover Mac OS X binary compatibility requirementscover the vagaries of Linux distros and other *nix systems

22 Chapter 5. Advanced Topics

Python Packaging Guide, Release

5.3.4 Publishing binary extensions

FIXME

cover publishing as wheel files on PyPI or a custom index servercover creation of Windows and Mac OS X installersmention the fact that Linux distros have a requirement to build fromsource in their own build systems anyway, so pre-built binaries for

*nix systems currently aren’t common

5.3. Binary Extensions 23

Python Packaging Guide, Release

24 Chapter 5. Advanced Topics

CHAPTER 6

Project Summaries

Page Status Complete

Last Reviewed 2014-01-22

Summaries and links for the most relevant projects in the space of Python installation and packaging.

6.1 pip

Docs | User list 1 | Dev list | Issues | Github | PyPI | irc:#pip

A tool for installing and managing Python packages.

6.2 virtualenv

Docs | User list | Dev list | Issues | Github | PyPI | irc:#pip

A tool for creating isolated Python environments.

6.3 setuptools

(includes easy_install)

Docs | User list 2 | Dev list | Issues | Bitbucket | PyPI | irc:#distutils

setuptools is a collection of enhancements to the Python distutils that allow you to more easily build and distributePython packages, especially ones that have dependencies on other packages.

distribute was a fork of setuptools that was recently merged back into setuptools (in v0.7), thereby making setuptoolsthe primary choice for Python packaging.

6.4 wheel

Docs | Mailing list 2 | Issues | Bitbucket | PyPI

1 pip was created by the same developer as virtualenv, and early on adopted the virtualenv mailing list, and it’s stuck ever since.2 Multiple projects reuse the distutils-sig mailing list as their user list.

25

Python Packaging Guide, Release

Primarily, the wheel project offers the bdist_wheel setuptools extension for creating wheel distributions. Addi-tionally, it offers it’s own command line utility for creating and installing wheels.

6.5 distlib

Docs | Mailing list 2 | Issues | Bitbucket | PyPI

Distlib is a library which implements low-level functions that relate to packaging and distribution of Python software.It consists in part of the functions in the packaging Python package, which was intended to be released as part ofPython 3.3, but was removed shortly before Python 3.3 entered beta testing.

6.6 Warehouse

Docs | Mailing list 2 | Issues | Github | irc:#warehouse

The new unreleased PyPI application (to replace the current site at https://pypi.python.org/pypi).

6.7 buildout

Docs | Mailing list 2 | Issues | PyPI | irc:#buildout

Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, someof which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.

6.8 bento

Docs | Mailing list | Issues | Github | PyPI

Bento is a packaging tool solution for python software, targeted as an alternative to distutils, setuptools, distribute,etc.... Bento’s philosophy is reproducibility, extensibility and simplicity (in that order).

6.9 conda

Docs

conda is an installation tool for managing Anaconda installations. Anaconda is a collection of powerful packagesfor Python that enables large-scale data management, analysis, and visualization for Business Intelligence, ScientificAnalysis, Engineering, Machine Learning, and more.

6.10 Hashdist

Docs | Github

Hashdist is a library for building non-root software distributions. Hashdist is trying to be “the Debian of choice forcases where Debian technology doesn’t work”. The best way for Pythonistas to think about Hashdist may be a morepowerful hybrid of virtualenv and buildout.

26 Chapter 6. Project Summaries

Python Packaging Guide, Release

6.10. Hashdist 27

Python Packaging Guide, Release

28 Chapter 6. Project Summaries

CHAPTER 7

PEP Summaries

Page Status Complete

Last Reviewed 2014-01-22

Summaries for the currently relevant PEPs in the space of Python installation and packaging.

7.1 PEP376 Database of Installed Python Distributions

PEP Link PEP376

PEP Status Accepted

Summary Describes a new .dist-info directory structure and system of metadata to be used wheninstalling distributions.

User Impact This has very little implication for users, except to be aware of the format if they go huntingthrough their site-packages.

Note: PEP426 will likely spawn a child PEP that updates this to use a json-based system ofmetadata.

Implementation Currently, only the bdist_wheel extension from the wheel project creates distribu-tions using this structure, although setuptools and pip are equipped to manage distributions installedin this way.

7.2 PEP425 Compatibility Tags for Built Distributions

PEP Link PEP425

PEP Status Accepted

Summary Specifies a tagging system to use in Built Distribution file names. The motivation for thesystem was to tag wheel distributions, which are covered in PEP427

Note: A revision to this PEP is likely due to simple tags like “linux_x86_64” not handling thevariation within linux platforms. Because of this, PyPI currently blocks uploading linux platform-specific wheels and pip won’t install linux platform-specific wheels from PyPI.

29

Python Packaging Guide, Release

User Impact As wheels become more common, users will notice the new tagging scheme in wheel file-names.

Implementation The bdist_wheel setuptools extensions generates wheels using this scheme, andpip’s wheel installer understands the scheme as of v1.4.

7.3 PEP427 The Wheel Binary Package Format 1.0

PEP Link PEP427

PEP Status Accepted

Summary Specifies a Built Distribution format, that is based on, but modernizes the Egg format. Wheelfilenames conform to PEP425

Note: PEP426 will likely spawn a child PEP that updates this to use a json-based system ofmetadata.

User Impact Built distributions are fast to install.

Implementation The bdist_wheel setuptools extension (available from wheel) generates wheels, andpip supports installing wheels as of v1.4.

7.4 PEP438 Transitioning to release-file hosting on PyPI

PEP Link PEP438

PEP Status Accepted

Summary Specifies a two-step plan to phase out the primary use of external download links on PyPI, forthe sake of security and installation speed.

User Impact pip (as of v1.5) will be faster and more secure by default.

Implementation Both pip and PyPI made changes during 2013 to implement this PEP.

7.5 PEP453 Explicit bootstrapping of pip in Python installations

PEP Link PEP453

PEP Status Accepted

Summary Proposes the inclusion of a method for explicitly bootstrapping pip as the default packagemanager for Python.

User Impact pip will be available in some Python installations without users having to install it.

Implementation The goal is to have this for Python 3.4. PEP453 includes an integration timeline.

7.6 PEP426 Metadata for Python Software Packages 2.0

PEP Link PEP426

PEP Status Draft

30 Chapter 7. PEP Summaries

Python Packaging Guide, Release

Summary Specifies version 2.0 of the metadata format. Version 1.0 is specified in PEP241. Version1.1 is specified in PEP314. Version 1.2 is specified in PEP345. This is a work in progress, andrepresents a major upgrade to the Packaging ecosystem. PEP440 is a child of this PEP, and morePEPs are likely to grow out of this, as it evolves.

User Impact When this is accepted, users themselves will not do anything to adopt the new system, butrather projects like pip, setuptools, and PyPI will make changes to conform to it, and then surfacenew features and functionality to users that are based on top of the new system.

Implementation Nothing at this time.

7.7 PEP440 Version Identification and Dependency Specification

PEP Link PEP440

PEP Status Draft

Summary Specifies a versioning system for Python projects that goes along with PEP426, and replacesPEP386. This system will be consistent with how most people version their projects today.

User Impact Once accepted, users will have a clear specification for what’s “correct” versioning forPython projects.

Implementation Nothing at this time.

7.8 PEP458 Surviving a Compromise of PyPI

PEP Link PEP458

PEP Status Draft

Summary Specifies an integration of PyPI with the “The Update Framework” (TUF).

User Impact pip will be more secure against various types of security attacks on PyPI and protect usersagainst them.

Implementation Nothing at this time.

7.7. PEP440 Version Identification and Dependency Specification 31

Python Packaging Guide, Release

32 Chapter 7. PEP Summaries

CHAPTER 8

A Packaging Timeline

Page Status Incomplete 1

Last Reviewed 2014-01-22

2013-06-09: The merge of setuptools and distribute was completed and released to PyPI. 2

2013-03-23: PyPA became the maintainer for the Python Packaging User Guide, which was forked from the “Hitch-hiker’s Guide to Packaging”. 3

2013-03-15: Packaging Dev and User Summits were held at Pycon 2013 to share ideas on the future of packaging. 4 5

2013-03-14: The intention to merge setuptools and distribute was announced by their respective maintainers, PJ Ebyand Jason Coombs. 6

2013-03-09: pip began depending on distlib. 7

2013-03-02: distlib began releasing to PyPI.

2013-03-17: PEP425 and PEP427 were accepted. Together, they specify a built-package format for Python calledWheel.

2012-06-19: The effort to include “Distutils2/Packaging” in Python 3.3 was abandoned due lack of involvement. 8

2008: distribute was forked from setuptools by Tarek Ziade, in an effort to create a more open project.

2008: pip was introduced by Ian Bicking as an alternative to easy_install (the installer included with setuptools)

2007: virtualenv was introduced by Ian Bicking, which allowed users to create isolated Python environments based ona central system installation of Python.

2006: buildout was introduced by Jim Fulton, with the goal to create a system for repeatable installations of potentiallycomplex projects.

2004: setuptools was introduced by Phillip Eby, which included the Egg format, and the ability to declare and auto-matically install dependencies.

2003: PyPI was up and running.

2002: Richard Jones started work on PyPI, and created PEP301 to describe it.

2001: PEP241 was written to standardize the metadata for packages.

1 What’s missing: 1) recent PEP438 events 2) D2 evolution and failure, 3) PEP453 approval, 4) buildout 2.0 not being isolated2 http://mail.python.org/pipermail/distutils-sig/2013-June/021160.html3 http://mail.python.org/pipermail/distutils-sig/2013-March/020224.html4 https://us.pycon.org/2013/community/openspaces/packaginganddistributionminisummit/5 http://www.pyvideo.org/video/1731/panel-directions-for-packaging6 http://mail.python.org/pipermail/distutils-sig/2013-March/020127.html7 https://github.com/pypa/pip/pull/8348 http://mail.python.org/pipermail/python-dev/2012-June/120430.html

33

Python Packaging Guide, Release

2000: catalog-sig was created to discuss creating a centralized index of packages.

2000: distutils was added to the Python standard library in Python 1.6.

1998: The distutils-sig dicussion list was created to discuss the development of distutils.

34 Chapter 8. A Packaging Timeline

CHAPTER 9

Glossary

Page Status Incomplete 1

Last Reviewed 2014-01-22

Binary Distribution A specific kind of Built Distribution that contains compiled extensions.

Built Distribution A Distribution format containing files and metadata that only need to be moved to the correctlocation on the target system, to be installed. Wheel is such a format, whereas distutil’s Source Distribution isnot, in that it requires a build step before it can be installed. This format does not imply that python files have tobe precompiled (Wheel intentionally does not include compiled python files).

Distribution A Python distribution is a versioned archive file that contains Python packages, modules, and otherresource files that are used to distribute a Release. The distribution file is what an end-user will download fromthe internet and install.

Distutils A standard package that comes with the Python Standard Library that has support for creating distributions.setuptools provides enhancements to distutils, and is more commonly used than distutils by itself.

Egg A Built Distribution format introduced by setuptools, which is being replaced by Wheel. For details, see TheInternal Structure of Python Eggs and Python Eggs

Extension Module A module written in the low-level language of the Python implementation: C/C++ for Python,Java for Jython. Typically contained in a single dynamically loadable pre-compiled file, e.g. a shared object(.so) file for Python extensions on Unix, a DLL (given the .pyd extension) for Python extensions on Windows,or a Java class file for Jython extensions.

Known Good Set (KGS) A set of distributions at specified versions which are compatible with each other. Typicallya test suite will be run which passes all tests before a specific set of packages is declared a known good set. Thisterm is commonly used by frameworks and toolkits which are comprised of multiple individual distributions.

Module The basic unit of code reusability in Python, existing in one of two types: Pure Module, or Extension Module.

Package (Meaning #1) A directory containing an __init__.py file (ex. mypackage/__init__.py), andalso usually containing modules (possibly along with other packages). You can import a package: importmypackage

Package (Meaning #2) A synonym for Distribution. It is common in Python to refer to a distribution using the term“package”. While the two meanings of the term “package” is not always 100% unambigous, the context of theterm “package” is usually sufficient to distinguish the meaning of the word. For example, the python installationtool pip is an acronym for “pip installs packages”. while technically the tool installs distributions. Even the sitewhere distributions are distributed at is called the “Python Package Index” (and not the “Python DistributionIndex”).

1 Although the current terms have been reviewed, there’s more terminology used by projects like pip and setuptools that needs to be added.

35

Python Packaging Guide, Release

Package Index A repository of distributions with a web interface to automate Distribution discovery and consump-tion.

Project A library, framework, script, plugin, application, or collection of data or other resources, or some combina-tion thereof.

Python projects must have unique names, which are registered on PyPI. Each project will then contain one ormore Releases, and each release may comprise one or more distributions.

Note that there is a strong convention to name a project after the name of the package that is imported to run thatproject. However, this doesn’t have to hold true. It’s possible to install a distribution from the project ‘spam’and have it provide a package importable only as ‘eggs’.

Pure Module A module written in Python and contained in a single .py file (and possibly associated .pyc and/or .pyofiles).

Python Packaging Authority (PyPA) PyPA is an informal working group that maintains some of the most relevantprojects for Python packaging. They host projects at https://github.com/pypa/ and https://bitbucket.org/pypa,and discuss issues at https://groups.google.com/forum/#!forum/pypa-dev.

Python Package Index (PyPI) PyPI is the default Package Index for the Python community. It is open to all Pythondevelopers to consume and distribute their distributions.

Release A snapshot of a Project at a particular point in time, denoted by a version identifier.

Making a release may entail the publishing of multiple Distributions. For example, if version 1.0 of a projectwas released, it could be available in both a source distribution format and a Windows installer distributionformat.

Source Archive An archive containing the raw source code for a Release, prior to creation of an Source Distributionor Built Distribution.

Source Distribution (or “sdist”) A distribution format (usually generated using python setup.py sdist)that provides metadata and the essential source files needed for installing by a tool like pip, or for generating aBuilt Distribution.

System Package A package provided in a format native to the operating system, e.g. an rpm or dpkg file.

Wheel A Built Distribution format introduced by PEP427 The Wheel Binary Package Format 1.0, which is intendedto replace the Egg format. Wheel is currently supported by pip.

Working Set A collection of distributions available for importing. These are the distributions that are on the sys.pathvariable. At most, one Distribution for a project is possible in a working set.

36 Chapter 9. Glossary