Spawning from matrix-org/synapse#14028 (comment)

The reason they failed before was because we're fighting against
stale caches across workers racing while waiting for invalidation,
see matrix-org/synapse#13185 (comment)

> Here is what happens:
>
>  1. `serverB` has `event1` stored as an `outlier` from previous requests (specifically from MSC3030 jump to date pulling in a missing `prev_event` after backfilling)
>  1. Client on `serverB` calls `/messages?dir=b`
>  1. `serverB:client_reader1` accepts the request and drives things
>  1. `serverB:client_reader1` has some backward extremities in range and requests `/backfill` from `serverA`
>  1. `serverB:client_reader1` processes the events from backfill including `event1` and puts them in the `_event_persist_queue`
>  1. `serverB:master` picks up the events from the `_event_persist_queue` and persists them to the database, de-outliers `event1` and invalidates its own cache and sends them over replication
>  1. `serverB:client_reader1` starts assembling the `/messages` response and gets `event1` out of the stale cache still as an `outlier`
>  1. `serverB:client_reader1` responds to the `/messages` request without `event1` because `outliers` are filtered out
>  1. `serverB:client_reader1` finally gets the replication data and invalidates its own cache for `event1` (too late, we already got the events from the stale cache and responded)

…rker

See #492 (comment)

See #492 (comment)