Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dependency management pattern #5946

Merged
merged 2 commits into from
Jun 3, 2019
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions docs/development/patterns.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,86 @@ Patterns
.. contents::
:local:

Dependency management
---------------------

Warehouse's approach to dependency management can be summarized as follows:

- Separate requirements files for different environments (deploy, development,
docs, linting, testing, etc.);
- All dependencies are pinned to precise versions, and include artifact hashes;
- Pinned requirements and subdependencies are compiled from ``.in`` files.

We install all dependencies with ``pip``, and we use |pip-tools|_ to compile
dependencies.

In practice, developers need to interact with our dependencies in three ways:

Upgrading existing dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dependencies are automatically upgraded via `Dependabot pull requests`_, and
occasionally merged by maintainers.

Adding new dependencies
~~~~~~~~~~~~~~~~~~~~~~~

Deciding to add a new dependency should be made carefully. Generally, we are
not opposed to adding more dependencies, however some effort should be made to
ensure that a given dependency:

- Is reasonably stable;
- Is currently maintained;
- Doesn't introduce a large amount of sub-dependencies.

All top-level dependencies are included in one or more ``.in`` files, which are
then compiled into ``.txt`` files with precise versions and artifact hashes.

When adding a new dependency, it's important to add it to the correct ``.in``
file:

=============== ============================================
File Purpose
=============== ============================================
``deploy.in`` Required only to run in production
``docs.in`` For our documentation
``lint.in`` For linting our docs and codebase
``main.in`` Every dependency of our web service
``tests.in`` Required to run our tests
=============== ============================================

Dependencies that are either private or aren't deployed to production aren't
compiled:

=============== ============================================
File Purpose
=============== ============================================
``dev.txt`` Various development dependencies
``ipython.txt`` Specific to using IPython as your shell
``theme.txt`` Private dependencies for our logos and theme
=============== ============================================

To add a new dependency:

1. Add the project name to the appropriate ``.in`` file
2. Recompile the dependencies for each modified ``.in`` file::

$ pip-compile --no-annotate --no-header --allow-unsafe --generate-hashes {file}.in

3. Commit the changes

Removing existing dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Only top-level dependencies should be removed. The process is similar to the
process for adding new dependencies:

1. Remove the project name from the appropriate ``.in`` file
2. Recompile the dependencies for each modified ``.in`` file::

$ pip-compile --no-annotate --no-header --allow-unsafe --generate-hashes {file}.in

3. Commit the changes

Returning vs Raising HTTP Exceptions
------------------------------------
Expand All @@ -30,3 +110,7 @@ Class Method
return.
``HTTPServerError`` (5xx) Raise
========================= ==================================

.. |pip-tools| replace:: ``pip-tools``
.. _pip-tools: https://pypi.org/project/pip-tools/
.. _Dependabot pull requests: https://github.com/pypa/warehouse/pulls?q=is%3Apr+is%3Aopen+label%3Adependencies