From 993cd110b599957e4fa313bc2b2c0fc33697915c Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Fri, 31 May 2019 17:09:53 -0500 Subject: [PATCH 1/2] Add dependency management pattern --- docs/development/patterns.rst | 84 +++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/docs/development/patterns.rst b/docs/development/patterns.rst index 67979268dc53..3d6287bf3a4d 100644 --- a/docs/development/patterns.rst +++ b/docs/development/patterns.rst @@ -4,6 +4,86 @@ Patterns .. contents:: :local: +Dependency management +--------------------- + +Warehouse's approach to dependency management can be summarized as follows: + +- Separate requirements files for different environments (deploy, development, + docs, linting, testing, etc.); +- All dependencies are pinned to precise versions, and include artifact hashes; +- Pinned requirements and subdependencies are compiled from ``.in`` files. + +We install all dependencies with ``pip``, and we use |pip-tools|_ to compile +dependencies. + +In practice, developers need to interact with our dependencies in three ways: + +Upgrading existing dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Dependencies are automatically upgraded via `Dependabot pull requests`_, and +occasionally merged by maintainers. + +Adding new dependencies +~~~~~~~~~~~~~~~~~~~~~~~ + +Deciding to add a new dependency should be made carefully. Generally, we are +not opposed to adding more dependencies, however some effort should be made to +ensure that a given dependency: + +- Is reasonably stable; +- Is currently maintained; +- Doesn't introduce a large amount of sub-dependencies. + +All top-level dependencies are included in one or more ``.in`` files, which are +then compiled into ``.txt`` files with precise versions and artifact hashses. + +When adding a new dependency, it's important to add it to the correct ``.in`` +file: + +=============== ============================================ +File Purpose +=============== ============================================ +``deploy.in`` Required only to run in production +``docs.in`` For our documentation +``lint.in`` For linting our docs and codebase +``main.in`` Every dependency of our web service +``tests.in`` Required to run our tests +=============== ============================================ + +Dependencies that are either private or aren't deployed to production aren't +compiled: + +=============== ============================================ +File Purpose +=============== ============================================ +``dev.txt`` Various development dependencies +``ipython.txt`` Specific to using IPython as your shell +``theme.txt`` Private dependencies for our logos and theme +=============== ============================================ + +To add a new dependency: + +1. Add the project name to the appropriate ``.in`` file +2. Recompile the dependencies for each modified ``.in`` file:: + + $ pip-compile --no-annotate --no-header --allow-unsafe --generate-hashes {file}.in + +3. Commit the changes + +Removing existing dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Only top-level dependencies should be removed. The process is similar to the +process for adding new dependencies: + +1. Remove the project name from the appropriate ``.in`` file +2. Recompile the dependencies for each modified ``.in`` file:: + + $ pip-compile --no-annotate --no-header --allow-unsafe --generate-hashes {file}.in + +3. Commit the changes Returning vs Raising HTTP Exceptions ------------------------------------ @@ -30,3 +110,7 @@ Class Method return. ``HTTPServerError`` (5xx) Raise ========================= ================================== + +.. |pip-tools| replace:: ``pip-tools`` +.. _pip-tools: https://pypi.org/project/pip-tools/ +.. _Dependabot pull requests: https://github.com/pypa/warehouse/pulls?q=is%3Apr+is%3Aopen+label%3Adependencies From 900167485332372236495afc37077abc60947004 Mon Sep 17 00:00:00 2001 From: Dustin Ingram Date: Mon, 3 Jun 2019 14:14:01 -0500 Subject: [PATCH 2/2] Fix typo --- docs/development/patterns.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/patterns.rst b/docs/development/patterns.rst index 3d6287bf3a4d..a9420ec2e266 100644 --- a/docs/development/patterns.rst +++ b/docs/development/patterns.rst @@ -37,7 +37,7 @@ ensure that a given dependency: - Doesn't introduce a large amount of sub-dependencies. All top-level dependencies are included in one or more ``.in`` files, which are -then compiled into ``.txt`` files with precise versions and artifact hashses. +then compiled into ``.txt`` files with precise versions and artifact hashes. When adding a new dependency, it's important to add it to the correct ``.in`` file: