Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clustersynchro: handle each of the pages in the resource list stage #591

Merged
merged 6 commits into from
Nov 28, 2023

Conversation

Iceber
Copy link
Member

@Iceber Iceber commented Nov 14, 2023

What type of PR is this?

/kind feature

What this PR does / why we need it:
In List, the paged pull method is used, so the resources within each paging will be processed directly.

  1. The resources in each page of the List are sent directly to the DeltaFIFO, and the ResourceHandler is notified to process the resources.
  2. Before sending to DeltaFIFO, the key of each resource is saved to the itemKeys list.
  3. after all paging pulls are complete, the resources are compared to those that will trigger the Delete event via DeltaFIFO.Replace(itemKeys).

Added two FeatureGates:
StreamHandlePaginatedListForResourceSync: stream handling of paged resources, resources within a pager will be processed as soon as possible instead of waiting until all resources are pulled before calling the ResourceHandler.
ForcePaginatedListForResourceSync: force paging, Reflector will sometimes use APIServer's cache, even if paging is specified APIServer will return all resources for performance, then it will skip Reflector's streaming memory optimization.

ForcePaginatedListForResourceSync may put extra pressure on the imported cluster APIServer, please be careful to enable it.

clustersynchro-manager add --page-size flags, you can control the page size of the List, the smaller the page size, the smaller the memory consumption, but may also generate more requests to APIServer.

Which issue(s) this PR fixes:
fix: #594

Special notes for your reviewer:

Does this PR introduce a user-facing change?:


@clusterpedia-bot
Copy link

Hi @Iceber,
Thanks for your pull request!
If the PR is ready, use the /auto-cc command to assign Reviewer to Review.
We will review it shortly.

Details

Instructions for interacting with me using comments are available here.
If you have questions or suggestions related to my behavior, please file an issue against the gh-ci-bot repository.

@clusterpedia-bot clusterpedia-bot added the kind/feature New feature label Nov 14, 2023
@Iceber Iceber changed the title clustersynchro: in the resource list stage, handle each of the pages [WIP] clustersynchro: in the resource list stage, handle each of the pages Nov 14, 2023
@Iceber Iceber changed the title [WIP] clustersynchro: in the resource list stage, handle each of the pages [WIP] clustersynchro: handle each of the pages in the resource list stage Nov 14, 2023
@Iceber Iceber force-pushed the reflector_with_result_stream branch 4 times, most recently from 07f4d15 to 202bb30 Compare November 17, 2023 02:09
@Iceber Iceber changed the title [WIP] clustersynchro: handle each of the pages in the resource list stage clustersynchro: handle each of the pages in the resource list stage Nov 17, 2023
@Iceber Iceber force-pushed the reflector_with_result_stream branch from 202bb30 to 298c60a Compare November 17, 2023 02:42
Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
Signed-off-by: Iceber Gu <caiwei95@hotmail.com>
@Iceber Iceber force-pushed the reflector_with_result_stream branch from c57cd32 to 2fc4da1 Compare November 28, 2023 03:17
@Iceber Iceber merged commit 9c2e456 into clusterpedia-io:main Nov 28, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reducing informer-reflector's memory footprint
2 participants