Test Synapse against Complement with workers.

[reivilibre]

Closes #12638 (once the tests pass, that is...).

For the time being (until #12848 is fixed), I've split them into 4 alphabetical groups.
I first went by dividing the alphabet up immediately, but then knocked together a quick script to give the best points to split based on runtime

SPLIT AT TestFilter
SPLIT AT TestMediaFilenames
SPLIT AT TestRoomCreate

Funnily enough, those are the quarter-points of the alphabet anyway (or 1 letter off).

It's not nice to need to do this, but it means we can enable Complement in CI right away until we can properly shave some time off. I have some ideas to do that, but they would definitely not be in scope for the same PR.

[reivilibre]

52 minutes! 😨

[richvdh]

I have reservations over this is a useful thing to do.

We are constrained as an organisation as to the number of GHA jobs we can run concurrently (to 60, according to https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits). Splitting the tests up this way does nothing to reduce the total amount of GHA worker time used by our tests, but does increase the chances of us monopolising the workers.

An alternative might be to exclude worker-mode-complement from the tests-done job, so we don't need to wait for it. But that's fiddly too.

le sigh.

[reivilibre]

Right, fair enough, I didn't realise we had this limit (well, I assumed there must be a limit, but I didn't know how much it was or how concerned we were about it).
You're right that this only reduces the wall time, at the cost of increasing the total consumed time (since each image has the overheads of building the Complement images, etc.).

Perhaps it may make sense, for now, to:

• restrict these worker tests to running on develop only(?) — gives us an indication if nothing else. Also prevents us having to 'wait' for it.
• try and get some speed improvements landed (e.g. the forking launcher, which cuts it down to ~26 minutes which is only a few minutes longer than the next slowest thing.)

Whilst testing, I've been cheating a bit and making Complement start before the lints are done; this gives it a few minutes headstart and then it's not the slowest thing on the list. If we're interested that much in shaving some time off, it could be worth considering, though I'd rather get some real time shavings if possible.

[richvdh]

Perhaps it may make sense, for now, to:

yeah, let's try what you suggest for now

[reivilibre]

Turns out you can't use a matrix in the if block of a job (YAML anchors are no good either), so I've duplicated it for now :/

[DMRobertson]

Turns out you can't use a matrix in the if block of a job (YAML anchors are no good either), so I've duplicated it for now :/

Are you sure about this? We have examples of doing so, e.g.

synapse/.github/workflows/latest_deps.yml

Line 58 in 57fac2a

if: ${{ matrix.postgres-version }}

[DMRobertson]

Oh, perhaps you mean the job-level if rather than the step-level if

[reivilibre]

Turns out you can't use a matrix in the if block of a job (YAML anchors are no good either), so I've duplicated it for now :/

Are you sure about this? We have examples of doing so, e.g.

synapse/.github/workflows/latest_deps.yml

Line 58 in 57fac2a

if: ${{ matrix.postgres-version }}

Yeah, see https://github.com/matrix-org/synapse/actions/runs/2414412187/workflow#L310

I don't think that example is using it at the same level with the same meaning (we want to skip a job, not a step)

[richvdh]

lgtm elsewise