Retention policy purge jobs fail if run_background_tasks_on is set to something other than master

[babolivier]

If run_background_tasks_on is set to anything other than master, Synapse will try to run background jobs on a worker, including the message retention policies support's purge jobs. However, these jobs end up calling the purge_history function of the storage layer (in the PurgeEventsStorage class) which doesn't exist on a worker, therefore the jobs fail with AttributeError: 'GenericWorkerSlavedStore' object has no attribute 'purge_history'.

This bug was introduced in #8513

cc @iansinclair

[clokep]

I guess we either need to move the method to the worker store or backout the changes that run that on a background worker. I think they're potentially heavy though, so if we could run it on a worker that would be best!

[babolivier]

Yes I tend to agree - though having something that fiddles with the database as much as purge_history does run on a worker sounds a bit scary to me.

[callahad]

Noting that @richvdh self-assigned: I believe you're a bit overloaded at the moment; would it make sense to hand this off to someone else (@DMRobertson?) with you providing review once impl'd?

[richvdh]

Noting that @richvdh self-assigned:

sorry, at one point it sounded like this was a drop-the-world level of urgency, but that turned out to be incorrect. Yes, it's unlikely I'll have time to look at this in the near future!

[callahad]

Let's tentatively try to address this after FOSDEM next year... noting that retention policies are still an experimental feature themselves.

[squahtx]

related: #11165

[Fred-Treebal]

Very interested in a fix on this issue, we run matrix under Kubernetes, running purges from the master creates high load cpu spikes for several minutes on each scheduled purge, I'd rather have those spikes on a non critical worker like backgroundtasks. thx.