Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Make scripts-dev pass mypy --disallow-untyped-defs #12356

Merged
merged 14 commits into from
Apr 27, 2022
Merged

Make scripts-dev pass mypy --disallow-untyped-defs #12356

merged 14 commits into from
Apr 27, 2022

Conversation

DMRobertson
Copy link
Contributor

@DMRobertson DMRobertson commented Apr 1, 2022 •
edited
Loading

I couldn't find a way to actually configure --disallow-untyped-defs in the scripts-dev section. I think this might be easier in pyproject.toml syntax rather than mypy.ini syntax; see also #11302 and #11315.

The release script was a bit fiddlier than I expected to get it typechecking, so that's broken down into smaller commits.

Follows on from #12355.

@DMRobertson DMRobertson force-pushed the dmr/typing-scripts-dev branch 4 times, most recently from 7d0e1bd to 31ba178 Compare April 8, 2022 16:44
David Robertson added 10 commits April 20, 2022 18:49
Typecheck federation_client.py
9310352
Typecheck sign_json
d955162
Easy type fixes for the release script
79a5ab9
Workaround a false positive about indexing into refs
42c7b52 
```
scripts-dev/release.py:495: error: Unsupported right operand type for in ("Callable[[], IterableList[Reference]]")  [operator]
scripts-dev/release.py:496: error: Value of type "Callable[[], IterableList[Reference]]" is not indexable  [index]
```

`refs` is an alias the the property `references`. Mypy gets confused by
the alias, see python/mypy#6700
Get tracking branch to avoid a second None check
c25e51d 
Mypy can't see that `tracking_branch()` returning truthy means that the
second call to `tracking_branch` returns a non-`None` value.

A use for the walrus operator?
Assert away a complaint about LegacyVersion
4cb3e76
Fix a complaint that click.edit can return None
b2fe52a
Fix complaint about writing a different type to release_branch
41618ef
Use set_reference over a write to .reference
dba3efc 
`.reference` is a property whose setter function is `set_reference`.
My isn't happy when we try to assign directly:

```
scripts-dev/release.py:208: error: Trying to assign name "reference" that is not in "__slots__" of type "git.refs.head.HEAD"  [misc]
scripts-dev/release.py:208: error: Incompatible types in assignment (expression has type "HEAD", variable has type "Union[Head, TagReference, RemoteReference, Reference]")  [assignment]
```

but calling `set_reference` ourselves makes the typechecker happy.

Confusingly the following three types are all different:

- the `refererence` property: `Union['Head', 'TagReference', 'RemoteReference', 'Reference']`
- return type of `_get_reference`: `SymbolicReference`
- `set_reference` first argument: `Union[Commit_ish, 'SymbolicReference', str]`
  where `Commit_ish = Union['Commit', 'TagObject', 'Blob', 'Tree']`

It seems that typecheckers haven't settled down on The Way to handle
properties whose getter and setter types differ: see e.g.
python/mypy#3004 .
Changelog
6b20e2d
@DMRobertson DMRobertson marked this pull request as ready for review April 22, 2022 14:17
@DMRobertson DMRobertson requested a review from a team as a code owner April 22, 2022 14:17
squahtx
squahtx reviewed Apr 22, 2022
View reviewed changes
Copy link
Contributor

@squahtx squahtx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, one question

scripts-dev/release.py Outdated
Comment on lines 304 to 306
edited_changes = click.edit(changes, require_save=False)
assert edited_changes is not None
changes = edited_changes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The click docs suggest that edited_changes would be None when the user closes the editor without saving: https://click.palletsprojects.com/en/7.x/utils/#launching-editors

Do we want to abort via an assert or use the unedited changes in that case?

Copy link
Contributor Author

@DMRobertson DMRobertson Apr 27, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the docs hard to read. It seems like there might be different behaviour if you're editing an existing file, as opposed to a temp file created to hold a string.

Testing with print(click.edit("Hello", require_save=...)) yielded:

require_save: False requre_save: True
quit without save original text None
save without changes original text original text
save with changes changed text changed text

So I think this assert is for mypy's benefit only. Perhaps it would be clearer to cast this to str with a comment?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious. In that case I'm happy with the assert!

scripts-dev/release.py Outdated Show resolved Hide resolved
@DMRobertson
Copy link
Contributor Author

@squahtx there were some merge conflicts to resolve after my poetry-related changes on the 1.58 release branch. Could you take another look please?

squahtx
squahtx approved these changes Apr 27, 2022
View reviewed changes
Copy link
Contributor

@squahtx squahtx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still happy.
What's the removal of the redbaron dependency about?

@DMRobertson
Copy link
Contributor Author

I'm still happy. What's the removal of the redbaron dependency about?

That comes from this commit on the 1.58 release branch. We used to use redbaron to parse __init__.py and update the version number in place. That's now handled by asking poetry to update pyproject.toml for us (from this commit on the release branch).

@DMRobertson DMRobertson enabled auto-merge (squash) April 27, 2022 13:00
@DMRobertson DMRobertson merged commit 30c8e7e into develop Apr 27, 2022
@DMRobertson DMRobertson deleted the dmr/typing-scripts-dev branch April 27, 2022 13:10
DMRobertson pushed a commit that referenced this pull request May 10, 2022
Merge tag 'v1.59.0rc1' into develop
2aad0ae 
Synapse 1.59.0rc1 (2022-05-10)
==============================

This release makes several changes that server administrators should be aware of:

- Device name lookup over federation is now disabled by default. ([\#12616](#12616))
- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](#12452), [\#12654](#12654))

See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.

Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](#12597))

Features
--------

- Support [MSC3266](matrix-org/matrix-spec-proposals#3266) room summaries over federation. ([\#11507](#11507))
- Implement [changes](matrix-org/matrix-spec-proposals@4a77139) to [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#12168](#12168), [\#12635](#12635), [\#12636](#12636), [\#12670](#12670))
- Extend the [module API](https://github.com/matrix-org/synapse/blob/release-v1.59/synapse/module_api/__init__.py) to allow modules to change actions for existing push rules of local users. ([\#12406](#12406))
- Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. ([\#12452](#12452))
- Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ([\#12654](#12654))
- Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid. ([\#12526](#12526))
- Implement [MSC3786](matrix-org/matrix-spec-proposals#3786): Add a default push rule to ignore `m.room.server_acl` events. ([\#12601](#12601))
- Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice. ([\#12619](#12619))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where the latest thread reply provided failed to include the proper bundled aggregations. ([\#12273](#12273))
- Fix a bug introduced in Synapse 1.22.0 where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper. ([\#12544](#12544))
- Fix a bug introduced in Synapse 1.57.0 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation. ([\#12570](#12570))
- Fix a long-standing bug where status codes would almost always get logged as `200!`, irrespective of the actual status code, when clients disconnect before a request has finished processing. ([\#12580](#12580))
- Fix race when persisting an event and deleting a room that could lead to outbound federation breaking. ([\#12594](#12594))
- Fix a bug introduced in Synapse 1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated. ([\#12633](#12633))
- Fix a long-standing bug where rooms containing power levels with string values could not be upgraded. ([\#12657](#12657))
- Prevent memory leak from reoccurring when presence is disabled. ([\#12656](#12656))

Updates to the Docker image
---------------------------

- Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments. ([\#12541](#12541))
- Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by @henryclw. ([\#12573](#12573))

Improved Documentation
----------------------

- Update SQL statements and replace use of old table `user_stats_historical` in docs for Synapse Admins. ([\#12536](#12536))
- Add missing linebreak to `pipx` install instructions. ([\#12579](#12579))
- Add information about the TCP replication module to docs. ([\#12621](#12621))
- Fixes to the formatting of `README.rst`. ([\#12627](#12627))
- Fix docs on how to run specific Complement tests using the `complement.sh` test runner. ([\#12664](#12664))

Deprecations and Removals
-------------------------

- Remove unstable identifiers from [MSC3069](matrix-org/matrix-spec-proposals#3069). ([\#12596](#12596))
- Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from
  [MSC2778](matrix-org/matrix-spec-proposals#2778). ([\#12597](#12597))
- Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk. ([\#12613](#12613))

Internal Changes
----------------

- Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time. ([\#12480](#12480))
- Immediately retry any requests that have backed off when a server comes back online. ([\#12500](#12500))
- Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests. ([\#12505](#12505))
- Consistently check if an object is a `frozendict`. ([\#12564](#12564))
- Protect module callbacks with read semantics against cancellation. ([\#12568](#12568))
- Improve comments and error messages around access tokens. ([\#12577](#12577))
- Improve docstrings for the receipts store. ([\#12581](#12581))
- Use constants for read-receipts in tests. ([\#12582](#12582))
- Log status code of cancelled requests as 499 and avoid logging stack traces for them. ([\#12587](#12587), [\#12663](#12663))
- Remove special-case for `twisted` logger from default log config. ([\#12589](#12589))
- Use `getClientAddress` instead of the deprecated `getClientIP`. ([\#12599](#12599))
- Add link to documentation in Grafana Dashboard. ([\#12602](#12602))
- Reduce log spam when running multiple event persisters. ([\#12610](#12610))
- Add extra debug logging to federation sender. ([\#12614](#12614))
- Prevent remote homeservers from requesting local user device names by default. ([\#12616](#12616))
- Add a consistency check on events which we read from the database. ([\#12620](#12620))
- Remove use of the `constantly` library and switch to enums for `EventRedactBehaviour`. Contributed by @andrewdoh. ([\#12624](#12624))
- Remove unused code related to receipts. ([\#12632](#12632))
- Minor improvements to the scripts for running Synapse in worker mode under Complement. ([\#12637](#12637))
- Move `pympler` back in to the `all` extras. ([\#12652](#12652))
- Fix spelling of `M_UNRECOGNIZED` in comments. ([\#12665](#12665))
- Release script: confirm the commit to be tagged before tagging. ([\#12556](#12556))
- Fix a typo in the announcement text generated by the Synapse release development script. ([\#12612](#12612))

- Fix scripts-dev to pass typechecking. ([\#12356](#12356))
- Add some type hints to datastore. ([\#12485](#12485))
- Remove unused `# type: ignore`s. ([\#12531](#12531))
- Allow unused `# type: ignore` comments in bleeding edge CI jobs. ([\#12576](#12576))
- Remove redundant lines of config from `mypy.ini`. ([\#12608](#12608))
- Update to mypy 0.950. ([\#12650](#12650))
- Use `Concatenate` to better annotate `_do_execute`. ([\#12666](#12666))
- Use `ParamSpec` to refine type hints. ([\#12667](#12667))
- Fix mypy against latest pillow stubs. ([\#12671](#12671))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@squahtx squahtx squahtx approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DMRobertson @squahtx