Add label to pending_workloads metric to show the inadmissible workloads

[alculquicondor]

What would you like to be added:

Add a label for the workloads that are inadmissible and are not being retried.

Why is this needed:

It's useful to understand the health of kueue and the cluster queues.
For example, if there are inadmissible workloads, the performance of the scheduler is not affected, but it might be an indication of a misconfiguration of the queues, or an attempt from a namespace to use resources that they shouldn't.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[alculquicondor]

/priority important-soon

/help

[k8s-ci-robot]

@alculquicondor:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/priority important-soon

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.