Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc(katib): update push-based metrics collector. #3844

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Electronic-Waste
Copy link
Member

This PR aims at guiding users using push-based metrics collection: kubeflow/katib#2340.

I rewrote the metrics-collector chapter and divided it into three parts:

  1. Overview: Introduce the type and the config of the metrics collector.
  2. Pull-based Metrics Collector: The original part of this chapter. I removed None MC.
  3. Push-based Metrics Collector: Describe the usage of Push MC, with an example attached.

cc👀 @kubeflow/wg-automl-leads . Please let me know if you have any suggestions.

Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sperlingxx for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@andreyvelich
Copy link
Member

Thanks for adding this @Electronic-Waste!

/assign @hbelmiro @StefanoFioravanzo

Copy link
Member Author

@Electronic-Waste Electronic-Waste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some explanations for the fields in tune function. If you have any questions, don't hesitate to contact me.

Copy link
Contributor

@hbelmiro hbelmiro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@google-oss-prow google-oss-prow bot added the lgtm label Sep 4, 2024
Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this. I left a few comments.
/assign @kubeflow/wg-training-leads

Use tune function and specify the `metrics_collector_config` field. You can reference to the following example:

```
import kubeflow.katib as katib
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use simple example here and remove all unnecessary parameters and function calls.

Copy link
Member Author

@Electronic-Waste Electronic-Waste Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreyvelich Sorry, I may not fully understand what you mean "unnecessary parameters and function calls".

I copied the example in the get-started chapter and replaced print with report_metrics(). Do you mean that I don't need to call the tune() function and just define a main function in the example?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, we should just show example how user can use tune API with push-based metrics collector without any additional parameters they should set. For example:

import kubeflow.katib as katib

def objective(parameters):
  import time
  import kubeflow.katib as katib
  time.sleep(5)
  result = 4 * int(parameters["a"])
  # Push metrics to Katib DB.
  katib.report_metrics({"result": result})

katib.KatibClient().tune(
  name="push-metrics-exp",
  objective=objective,
  parameters= {"a": katib.search.int(min=10, max=20)}
  objective_metric_name="result",
  max_trial_count=2,
  metrics_collector_config={"kind": "Push"},
  # When SDK is released, replace it with packages_to_install=["kubeflow-katib==0.18.0"] 
  packages_to_install=["git+https://github.com/kubeflow/katib.git@master#subdirectory=sdk/python/v1beta1"],
)

That should allow user to focus on important changes they need to make to try this out.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andreyvelich Thanks for your clarification! I'll update the blog.


- [`tune`](https://github.com/kubeflow/katib/blob/master/sdk/python/v1beta1/kubeflow/katib/api/katib_client.py#L166) function

Use tune function and specify the `metrics_collector_config` field. You can reference to the following example:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might need to explain how report_metrics() functions works separately. E.g. user needs to install kubeflow-katib SDK in their environment, and the report_metrics() will automatically add the timestamp for metrics.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. I'll add the explanation to report_metrics().

@google-oss-prow google-oss-prow bot removed the lgtm label Sep 4, 2024
Copy link

New changes are detected. LGTM label has been removed.

Signed-off-by: Electronic-Waste <2690692950@qq.com>
Signed-off-by: Electronic-Waste <2690692950@qq.com>
@Electronic-Waste
Copy link
Member Author

@kubeflow/wg-automl-leads I've made some changes to the PR as you request. PTAL👀 if you have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants