Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pageserver: generation number fetch on startup and use in /attach #5163

Merged
merged 46 commits into from
Sep 6, 2023
Merged

pageserver: generation number fetch on startup and use in /attach #5163

merged 46 commits into from
Sep 6, 2023

Commits on Aug 31, 2023

  1. libs: add Generation::previous

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    e839fd1 View commit details
    Browse the repository at this point in the history

  2. pageserver_api: add types for control plane calls

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    1cedf33 View commit details
    Browse the repository at this point in the history

  3. pageserver: optional generation input during create & attach

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    bf4572f View commit details
    Browse the repository at this point in the history

  4. pageserver: add control_plane_api conf

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    233afac View commit details
    Browse the repository at this point in the history

  5. pageserver: call into control plane on startup

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    ac8ef22 View commit details
    Browse the repository at this point in the history

  6. pageserver: if control plane API is disabled, ignore generations

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    44e6125 View commit details
    Browse the repository at this point in the history

  7. pageserver: generation-aware index_part.json loading

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    38d5dec View commit details
    Browse the repository at this point in the history

  8. control_plane: add attachement_service, a mock control plane

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    885537a View commit details
    Browse the repository at this point in the history

  9. neon_local: manage attachment_service

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    08fac67 View commit details
    Browse the repository at this point in the history

  10. control_plane: implement attach hook

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    5e49aa5 View commit details
    Browse the repository at this point in the history

  11. remote_storage: fix LocalFs list_files

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    2ebe12e View commit details
    Browse the repository at this point in the history

  12. tests: enable generations in neon_fixture

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    9deaa1b View commit details
    Browse the repository at this point in the history

  13. clippy: permit many args to schedule_local_tenant_processing 

    When we refactor mgr.rs to implement a Mgr type, this arg
count will go down.
    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    af28cd1 View commit details
    Browse the repository at this point in the history

  14. Refactor TenantId encoding in JSON APIs

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    acce508 View commit details
    Browse the repository at this point in the history

  15. clippy

    @jcsp
    jcsp committed Aug 31, 2023
    Configuration menu
    Copy the full SHA
    1856d75 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. Update pageserver/src/tenant/remote_timeline_client/download.rs 

    Co-authored-by: Christian Schwarz <christian@neon.tech>
    @jcsp @problame
    jcsp and problame committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    e2bf591 View commit details
    Browse the repository at this point in the history

  2. Update pageserver/src/tenant/remote_timeline_client.rs 

    Co-authored-by: Christian Schwarz <christian@neon.tech>
    @jcsp @problame
    jcsp and problame committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    fd2ea25 View commit details
    Browse the repository at this point in the history

  3. Update pageserver/src/tenant/remote_timeline_client.rs 

    Co-authored-by: Christian Schwarz <christian@neon.tech>
    @jcsp @problame
    jcsp and problame committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    5fc0874 View commit details
    Browse the repository at this point in the history

  4. More consistent handling of generations in pageserver API 

    - Create gives a 400 if generation is omitted and control_plane_api is
  set
- Load can take one now
    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    891f7c4 View commit details
    Browse the repository at this point in the history

  5. clippy

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    8ea760a View commit details
    Browse the repository at this point in the history

  6. refactor control plane client

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    4d6b89d View commit details
    Browse the repository at this point in the history

  7. Cancellation and exponential backoff in control plane re-attach

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    9a5899c View commit details
    Browse the repository at this point in the history

  8. Fix Generation::previous and refactor suffix parsing

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    2649114 View commit details
    Browse the repository at this point in the history

  9. More efficient .max()

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    a983485 View commit details
    Browse the repository at this point in the history

  10. Tighten up download_index_part 

    Various comments, and properly cover the case of loading a stale
generation where an index already exists in our own generation.
    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    c1ca773 View commit details
    Browse the repository at this point in the history

  11. remove unused constructor arg

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    9dc2f72 View commit details
    Browse the repository at this point in the history

  12. Apply suggestions from code review 

    Co-authored-by: Christian Schwarz <christian@neon.tech>
    @jcsp @problame
    jcsp and problame committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    bb278aa View commit details
    Browse the repository at this point in the history

  13. Handle control plane URLs without a trailing slash

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    fb36329 View commit details
    Browse the repository at this point in the history

  14. Fix python lint complaint

    @jcsp
    jcsp committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    2553e48 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. Merge remote-tracking branch 'upstream/main' into jcsp/generation-num… 

    …bers-pt2
    @jcsp
    jcsp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    1cea6fb View commit details
    Browse the repository at this point in the history

  2. fixup merge

    @jcsp
    jcsp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    2af98aa View commit details
    Browse the repository at this point in the history

  3. Use safe tenant dir deletion in /re-attach handling

    @jcsp
    jcsp committed Sep 4, 2023
    Configuration menu
    Copy the full SHA
    a88fa7d View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. Update pageserver/src/tenant/remote_timeline_client/download.rs 

    Co-authored-by: Joonas Koivunen <joonas@neon.tech>
    @jcsp @koivunej
    jcsp and koivunej committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    85c737e View commit details
    Browse the repository at this point in the history

  2. Apply suggestions from code review 

    Co-authored-by: Joonas Koivunen <joonas@neon.tech>
    @jcsp @koivunej
    jcsp and koivunej committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    03c5ab0 View commit details
    Browse the repository at this point in the history

  3. Cleanup of attachment_service binary

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    382a930 View commit details
    Browse the repository at this point in the history

  4. improve logging in index part download

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    008545c View commit details
    Browse the repository at this point in the history

  5. libs: flatten a match{}

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    864fb74 View commit details
    Browse the repository at this point in the history

  6. flatten a match{}

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    32548c7 View commit details
    Browse the repository at this point in the history

  7. Use DisplayFromStr instead of hex

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    bc32ec6 View commit details
    Browse the repository at this point in the history

  8. Add track_caller to functions that can panic

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    6467853 View commit details
    Browse the repository at this point in the history

  9. clippy

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    386542e View commit details
    Browse the repository at this point in the history

  10. Add a log span to a unit test

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    8837479 View commit details
    Browse the repository at this point in the history

  11. Merge remote-tracking branch 'upstream/main' into jcsp/generation-num… 

    …bers-pt2
    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    966b5e3 View commit details
    Browse the repository at this point in the history

  12. pageserver: fix layer reconciliation with generations 

    This was wrongly assuming generations should be the same: the local
metadata will actually always have the current generation set, and
in this situation we want to UseLocal if the sizes match, but take
the metadata (generation) from the remote metadata.
    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    606a392 View commit details
    Browse the repository at this point in the history

  13. Apply suggestions from code review 

    Co-authored-by: Joonas Koivunen <joonas@neon.tech>
    @jcsp @koivunej
    jcsp and koivunej committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    db3dc06 View commit details
    Browse the repository at this point in the history

  14. unwrap_or_else

    @jcsp
    jcsp committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    562e47e View commit details
    Browse the repository at this point in the history