Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Move pympler back into the all extras #12652

Merged
merged 2 commits into from
May 7, 2022
Merged

Conversation

DMRobertson
Copy link
Contributor

Undoes a change I made in #12381. I can't fully remember my reasoning,
but this changed the contents of the debian packages in a backwards
incompatible way. We're not aware of anyone who's been bitten by this,
but we still want to fix it.

To the reviewer: please be convinced that the debian packages will still
contain pympler after this change.

Undoes a change I made in #12381. I can't fully remember my reasoning,
but this changed the contents of the debian packages in a backwards
incompatible way. We're not aware of anyone who's been bitten by this,
but we still want to fix it.

To the reviewer: please be convinced that the debian packages will still
contain pympler after this change.
@DMRobertson DMRobertson requested a review from a team as a code owner May 6, 2022 12:45
@DMRobertson DMRobertson linked an issue May 6, 2022 that may be closed by this pull request
Copy link
Contributor

@H-Shay H-Shay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am convinced.

@DMRobertson DMRobertson merged commit 0ce2201 into develop May 7, 2022
@DMRobertson DMRobertson deleted the dmr/packaging-pympler branch May 7, 2022 12:40
# redis
"txredisapi", "hiredis",
# cache_memory
"pympler",
Copy link
Contributor

@MadLittleMods MadLittleMods May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to break the Docker builds in complement.sh in some way I don't understand.

Here is a git bisect of sorts:

❌ d80a7ab151afd6919b4e14a258105ab59146d528

❌ 0ce2201932488e98cab8a7d81788e5bcf8f8dd5e

✅ 051a1c3f220938a0ea1a5b328c268bdb3d1ad592

✅ 2607b3e1816341b3b8534077bd5d3a4daf3a3d15

✅ c2d50e9f6c5f7b01cbd8bf1dca36cb8c0e7b007f

✅ 07fa53ec40106b97ba2c12a2dcc5446325e5fb61

And how it breaks:

Relevant part of the full-run:

#18 2.864 ERROR: Could not find a version that satisfies the requirement canonicaljson==1.6.0 (from versions: 1.0.0, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.2.0, 1.3.0, 1.4.0, 1.5.0)
#18 2.864 ERROR: No matching distribution found for canonicaljson==1.6.0

Full-run:

$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestSendAndFetchMessage
[+] Building 4.1s (17/24)
 => [internal] load build definition from Dockerfile                                                                                                             0.0s
 => => transferring dockerfile: 37B                                                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                                                0.0s
 => => transferring context: 35B                                                                                                                                 0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                       0.3s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:443aab4ca21183e069e7d8b2dc68006594f40bddf1b15bbd83f5137bd93e80e2                                  0.0s
 => [internal] load build definition from Dockerfile                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                               0.3s
 => [internal] load build context                                                                                                                                0.1s
 => => transferring context: 259.54kB                                                                                                                            0.1s
 => [requirements 1/6] FROM docker.io/library/python:3.9-slim@sha256:ba3b77ddbc953cdb8d998b2052088d4af4b8805805e5b01975a05af4e19855ea                            0.0s
 => CACHED [requirements 2/6] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get   0.0s
 => CACHED [requirements 3/6] RUN --mount=type=cache,target=/root/.cache/pip   pip install --user git+https://github.com/python-poetry/poetry.git@fb13b3a676f47  0.0s
 => CACHED [requirements 4/6] WORKDIR /synapse                                                                                                                   0.0s
 => CACHED [requirements 5/6] COPY pyproject.toml poetry.lock /synapse/                                                                                          0.0s
 => CACHED [requirements 6/6] RUN /root/.local/bin/poetry export --extras all -o /synapse/requirements.txt                                                       0.0s
 => CACHED [builder 2/7] RUN    --mount=type=cache,target=/var/cache/apt,sharing=locked    --mount=type=cache,target=/var/lib/apt,sharing=locked  apt-get updat  0.0s
 => CACHED [builder 3/7] COPY --from=requirements /synapse/requirements.txt /synapse/                                                                            0.0s
 => ERROR [builder 4/7] RUN --mount=type=cache,target=/root/.cache/pip   pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requir  3.0s
------
 > [builder 4/7] RUN --mount=type=cache,target=/root/.cache/pip   pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requirements.txt:
#18 2.309 Ignoring importlib-metadata: markers 'python_full_version >= "3.7.1" and python_version < "3.8"' don't match your environment
#18 2.310 Ignoring importlib-resources: markers 'python_full_version >= "3.7.1" and python_version < "3.9"' don't match your environment
#18 2.315 Ignoring psycopg2cffi-compat: markers 'python_full_version >= "3.7.1" and python_full_version < "4.0.0" and platform_python_implementation == "PyPy"' don't match your environment
#18 2.316 Ignoring psycopg2cffi: markers 'python_full_version >= "3.7.1" and python_full_version < "4.0.0" and platform_python_implementation == "PyPy"' don't match your environment
#18 2.320 Ignoring twisted-iocpsupport: markers 'python_full_version >= "3.7.1" and python_full_version < "4.0.0" and platform_system == "Windows"' don't match your environment
#18 2.321 Ignoring zipp: markers 'python_full_version >= "3.7.1" and python_version < "3.8" or python_full_version >= "3.7.1" and python_version < "3.9"' don't match your environment
#18 2.444 Collecting attrs==21.4.0
#18 2.446   Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
#18 2.495 Collecting authlib==0.15.5
#18 2.498   Using cached Authlib-0.15.5-py2.py3-none-any.whl (203 kB)
#18 2.542 Collecting automat==20.2.0
#18 2.544   Using cached Automat-20.2.0-py2.py3-none-any.whl (31 kB)
#18 2.650 Collecting bcrypt==3.2.0
#18 2.652   Using cached bcrypt-3.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (61 kB)
#18 2.712 Collecting bleach==4.1.0
#18 2.715   Using cached bleach-4.1.0-py2.py3-none-any.whl (157 kB)
#18 2.864 ERROR: Could not find a version that satisfies the requirement canonicaljson==1.6.0 (from versions: 1.0.0, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.2.0, 1.3.0, 1.4.0, 1.5.0)
#18 2.864 ERROR: No matching distribution found for canonicaljson==1.6.0
------
executor failed running [/bin/sh -c pip install --prefix="/install" --no-deps --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1

Copy link
Contributor

@MadLittleMods MadLittleMods May 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right when I have gathered all of the info and the results were consistent for all my testing, it started working (even on the latest develop). I'm assuming this is some Docker build caching layer problem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caching would be my guess, or maybe a temporary network failure that means the build process can't contact PyPI. I'm happy to investigate if you see it again though!

DMRobertson pushed a commit that referenced this pull request May 10, 2022
Synapse 1.59.0rc1 (2022-05-10)
==============================

This release makes several changes that server administrators should be aware of:

- Device name lookup over federation is now disabled by default. ([\#12616](#12616))
- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](#12452), [\#12654](#12654))

See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.

Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](#12597))

Features
--------

- Support [MSC3266](matrix-org/matrix-spec-proposals#3266) room summaries over federation. ([\#11507](#11507))
- Implement [changes](matrix-org/matrix-spec-proposals@4a77139) to [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#12168](#12168), [\#12635](#12635), [\#12636](#12636), [\#12670](#12670))
- Extend the [module API](https://github.com/matrix-org/synapse/blob/release-v1.59/synapse/module_api/__init__.py) to allow modules to change actions for existing push rules of local users. ([\#12406](#12406))
- Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. ([\#12452](#12452))
- Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ([\#12654](#12654))
- Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid. ([\#12526](#12526))
- Implement [MSC3786](matrix-org/matrix-spec-proposals#3786): Add a default push rule to ignore `m.room.server_acl` events. ([\#12601](#12601))
- Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice. ([\#12619](#12619))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where the latest thread reply provided failed to include the proper bundled aggregations. ([\#12273](#12273))
- Fix a bug introduced in Synapse 1.22.0 where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper. ([\#12544](#12544))
- Fix a bug introduced in Synapse 1.57.0 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation. ([\#12570](#12570))
- Fix a long-standing bug where status codes would almost always get logged as `200!`, irrespective of the actual status code, when clients disconnect before a request has finished processing. ([\#12580](#12580))
- Fix race when persisting an event and deleting a room that could lead to outbound federation breaking. ([\#12594](#12594))
- Fix a bug introduced in Synapse 1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated. ([\#12633](#12633))
- Fix a long-standing bug where rooms containing power levels with string values could not be upgraded. ([\#12657](#12657))
- Prevent memory leak from reoccurring when presence is disabled. ([\#12656](#12656))

Updates to the Docker image
---------------------------

- Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments. ([\#12541](#12541))
- Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by @henryclw. ([\#12573](#12573))

Improved Documentation
----------------------

- Update SQL statements and replace use of old table `user_stats_historical` in docs for Synapse Admins. ([\#12536](#12536))
- Add missing linebreak to `pipx` install instructions. ([\#12579](#12579))
- Add information about the TCP replication module to docs. ([\#12621](#12621))
- Fixes to the formatting of `README.rst`. ([\#12627](#12627))
- Fix docs on how to run specific Complement tests using the `complement.sh` test runner. ([\#12664](#12664))

Deprecations and Removals
-------------------------

- Remove unstable identifiers from [MSC3069](matrix-org/matrix-spec-proposals#3069). ([\#12596](#12596))
- Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from
  [MSC2778](matrix-org/matrix-spec-proposals#2778). ([\#12597](#12597))
- Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk. ([\#12613](#12613))

Internal Changes
----------------

- Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time. ([\#12480](#12480))
- Immediately retry any requests that have backed off when a server comes back online. ([\#12500](#12500))
- Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests. ([\#12505](#12505))
- Consistently check if an object is a `frozendict`. ([\#12564](#12564))
- Protect module callbacks with read semantics against cancellation. ([\#12568](#12568))
- Improve comments and error messages around access tokens. ([\#12577](#12577))
- Improve docstrings for the receipts store. ([\#12581](#12581))
- Use constants for read-receipts in tests. ([\#12582](#12582))
- Log status code of cancelled requests as 499 and avoid logging stack traces for them. ([\#12587](#12587), [\#12663](#12663))
- Remove special-case for `twisted` logger from default log config. ([\#12589](#12589))
- Use `getClientAddress` instead of the deprecated `getClientIP`. ([\#12599](#12599))
- Add link to documentation in Grafana Dashboard. ([\#12602](#12602))
- Reduce log spam when running multiple event persisters. ([\#12610](#12610))
- Add extra debug logging to federation sender. ([\#12614](#12614))
- Prevent remote homeservers from requesting local user device names by default. ([\#12616](#12616))
- Add a consistency check on events which we read from the database. ([\#12620](#12620))
- Remove use of the `constantly` library and switch to enums for `EventRedactBehaviour`. Contributed by @andrewdoh. ([\#12624](#12624))
- Remove unused code related to receipts. ([\#12632](#12632))
- Minor improvements to the scripts for running Synapse in worker mode under Complement. ([\#12637](#12637))
- Move `pympler` back in to the `all` extras. ([\#12652](#12652))
- Fix spelling of `M_UNRECOGNIZED` in comments. ([\#12665](#12665))
- Release script: confirm the commit to be tagged before tagging. ([\#12556](#12556))
- Fix a typo in the announcement text generated by the Synapse release development script. ([\#12612](#12612))

- Fix scripts-dev to pass typechecking. ([\#12356](#12356))
- Add some type hints to datastore. ([\#12485](#12485))
- Remove unused `# type: ignore`s. ([\#12531](#12531))
- Allow unused `# type: ignore` comments in bleeding edge CI jobs. ([\#12576](#12576))
- Remove redundant lines of config from `mypy.ini`. ([\#12608](#12608))
- Update to mypy 0.950. ([\#12650](#12650))
- Use `Concatenate` to better annotate `_do_execute`. ([\#12666](#12666))
- Use `ParamSpec` to refine type hints. ([\#12667](#12667))
- Fix mypy against latest pillow stubs. ([\#12671](#12671))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The all extras package group no longer includes pympler
3 participants