pageserver: deletion queue & generation validation for deletions

[jcsp]

Problem

Pageservers must not delete objects or advertise updates to remote_consistent_lsn without checking that they hold the latest generation for the tenant in question (see the RFC)

In this PR:

• A new "deletion queue" subsystem is introduced, through which deletions flow
• RemoteTimelineClient is modified to send deletions through the deletion queue:
  • For GC & compaction, deletions flow through the full generation verifying process
  • For timeline deletions, deletions take a fast path that bypasses generation verification
• The last_uploaded_consistent_lsn value in UploadQueue is replaced with a mechanism that maintains a "projected" lsn (equivalent to the previous property), and a "visible" LSN (which is the one that we may share with safekeepers).
• Until control_plane_api is set, all deletions skip generation validation
• Tests are introduced for the new functionality in test_pageserver_generations.py

Once this lands, if a pageserver is configured with the control_plane_api configuration added in #5163, it becomes safe to attach a tenant to multiple pageservers concurrently.

Summary of changes

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2520 tests run: 2403 passed, 0 failed, 117 skipped (full report)

---

Flaky tests (5)

Postgres 16

• test_crafted_wal_end[last_wal_record_xlog_switch_ends_on_page_boundary]: release
• test_partial_evict_tenant: release
• test_many_timelines: release
• test_wal_lagging: release

Postgres 14

• test_get_tenant_size_with_multiple_branches: release

Code coverage (full report)

• functions: 52.9% (8018 of 15145 functions)
• lines: 81.2% (47003 of 57884 lines)

---

_{The comment gets automatically updated with the latest test results
8af4131 at 2023-09-26T15:05:57.985Z :recycle:}

[koivunej]

This is a big chunk but let's handle rest of changes if any after merging.

[problame]

Re-reviewed validation & recovery.

I'm having trouble keeping track of error handling correctness in validator.rs when we write to disk.
At the offsite, we agreed that local IO error should crash the pageserver.
Started Slack thread about that.

Other than that, only few small remarks.

[problame]

We agreed in Slack that there will be a follow-up PR that will make the pageserver process abort/panic/exit if there are any local filesystem IO errors encountered by deletion queue.

With that, I'm OK with merging this PR.

(I still dislike the over-use of the term "flush", but, let's take that to a follow-up if any.)

[jcsp]

This hit a test failure that didn't have an existing ticket: #5385 -- I looked at the test and I don't think it's touching anything in that changed in this PR, and the test also looks like it might be racy by not waiting for pageserver ingest after it writes to the database. Will follow up separately.

[koivunej]

(re-ran failed)