Consider supporting SuccessPolicy and FailurePolicy

[terrytangyuan]

We recently added SuccessPolicy in tf-operator kubeflow/training-operator#1165 and are considering adding FailurePolicy to handle the case of failure in kubeflow/training-operator#1170. Once it's mature and if we see a common pattern in other operators, we should consider moving that to kubeflow/common.

cc @gaocegege @Jeffwan @johnugeorge @ChanYiLin @pingsutw

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.77
area/operator	0.85

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[kf-label-bot-dev]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.77

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[Jeffwan]

Having success/failure would be great which would be easier for different frameworks to handle errors and it help make reconciler logic extensible.

[zw0610]

With fault-tolerant & elastic distributed training propagating among more frameworks, a universal definition of failure and success for a distributed training job shall benefit developers for clarifying logic when handling pods failed or recently joined.