Model training job fails immediately

[rpallikonda]

I downloaded the repo, and completed the self hosting of the FMLTC app on personal google account. I couldn't get the model training job to get executed, it immediately fails. Would appreciate pointers to triage the same, as I cannot proceed further.

[lizlooney]

Can you check the app engine logs? That would be at console.cloud.google.com/logs/viewer?project=<your_project>&resource=gae_app
Look for the logs around the /startTrainingModel.

One thing to note is that it appears that it couldn't even create the training job, because the error occurred while the dialog box was still up. That's unusual and makes me suspect that something went wrong during the project setup.

[cmacfarl]

Another thing to look at is if there any errors in the browser console.

[rpallikonda]

Hi @lizlooney @cmacfarl Thank you for the replies. Pasting the log message below (scrubbed the project and service account details).

Please suggest what could be stopping the gcloud jobs to be erroring on quota.

The ml createJob is failing the Quota failure as pasted below. I have activated my AI platform training and prediction API quota as well. Please see the screenshot below.

{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 8,
"message": "Quota failure for project . The request for 1 K80 accelerators exceeds the allowed maximum of 0 A100, 0 K80, 0 P100, 0 P4, 0 T4, 0 TPU_V2, 0 TPU_V2_POD, 0 TPU_V3, 0 TPU_V3_POD, 0 V100 accelerators. To read more about Cloud ML Engine quota, see https://cloud.google.com/ml-engine/quotas."
},
"authenticationInfo": {
"principalEmail": "",
"serviceAccountKeyName": "//iam.googleapis.com/projects/"
},
"requestMetadata": {
"callerIp": "2600:1900:2000:66:400::d",
"callerSuppliedUserAgent": "(gzip),gzip(gfe)",
"requestAttributes": {
"time": "2021-11-14T03:15:20.158430Z",
"auth": {}
},
"destinationAttributes": {}
},
"serviceName": "ml.googleapis.com",
"methodName": "google.cloud.ml.v1.JobService.CreateJob",
"authorizationInfo": [
{
"resource": "projects/",
"permission": "ml.jobs.create",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/",
"request": {
"job": {
"jobId": "eval_2cf806681eab42ed914cc3d7bb15e1dd",
"trainingInput": {
"packageUris": [
"gs:///static/training/object_detection-0.1_2.5.0.tar.gz",
"gs:///static/training/slim-0.1.tar.gz",
"gs:///static/training/pycocotools-2.0.tar.gz"
],
"scaleTier": "BASIC_GPU",
"jobDir": "gs://-blobs/1c1c1e2b91ec40f6a4dd08c7c536c622/models/2cf806681eab42ed914cc3d7bb15e1dd",
"pythonModule": "object_detection.model_main_tf2",
"pythonVersion": "3.7",
"args": [
"--model_dir",
"gs://-blobs/1c1c1e2b91ec40f6a4dd08c7c536c622/models/2cf806681eab42ed914cc3d7bb15e1dd",
"--pipeline_config_path",
"gs://*****-blobs/1c1c1e2b91ec40f6a4dd08c7c536c622/models/2cf806681eab42ed914cc3d7bb15e1dd/pipeline.config",
"--checkpoint_dir",
"gs://-blobs/1c1c1e2b91ec40f6a4dd08c7c536c622/models/2cf806681eab42ed914cc3d7bb15e1dd"
],
"runtimeVersion": "2.5",
"region": "us-central1"
}
},
"@type": "type.googleapis.com/google.cloud.ml.v1.CreateJobRequest",
"parent": "projects/"
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
}
},
"insertId": "120pkpcc1r7",
"resource": {
"type": "audited_resource",
"labels": {
"method": "google.cloud.ml.v1.JobService.CreateJob",
"project_id": "",
"service": "ml.googleapis.com"
}
},
"timestamp": "2021-11-14T03:15:20.145535Z",
"severity": "ERROR",
"logName": "projects/*/logs/cloudaudit.googleapis.com%2Factivity",
"receiveTimestamp": "2021-11-14T03:15:20.939453673Z"
}

[cmacfarl]

That's interesting. The json message property there indicates that google thinks you have zero quota for any type of GPU or TPU.

If you go to the main ml.googleapis.com page is your Activation status "Enabled"?

https://console.cloud.google.com/apis/api/ml.googleapis.com/overview?project='your-project'

[rpallikonda]

Hi @cmacfarl,
Yes, I have it enabled. I tried few options to request quota without much success. Looks like it is not given to newbie projects like mine.

Is there any other hosted fmltc app that I could be able to use to train one simple model, and get unblocked?

[lizlooney]

Can you double-check that you performed all the items in the "Grant the ml.serviceAgent role to your TPU service account" section of the readme?

[cmacfarl]

That's an interesting theory and if he has then there should be a "service-*@cloud-tpu.iam.gserviceaccount.com" email in the Principal column on the main IAM page.

But, the json message explicitly says he's asking for 1 K80 accelerator, which might imply that it thinks he's using GPUs instead of TPUs. Is a TPU service agent still required for GPU usage?

[rpallikonda]

Hi @lizlooney I do have the ml.serviceAgent role configured to the TPU service account. Attaching the IAM policies

I can provide additional info, to help with triaging the issue

[cmacfarl]

Not that it really matters given the error message Google is returning, but the default config for the repo is to use TPU instances and yet your log indicates it's trying to use a GPU to train. Did you change any of the software to override the default or did you set a use_tpu configuration item to false?

Is there any other hosted fmltc app that I could be able to use to train one simple model, and get unblocked?

Are you associated with an FTC team and do you have a FIRST dashboard account?

[rpallikonda]

@cmacfarl I didn't change any code. Followed the setup instructions to the dot.

Yes, I am associated with a FTC team (team# 18307). I do have a FIRST dashboard account.

[rpallikonda]

@cmacfarl Please do share instructions that would help me to access a shared installation/deployment of the FMLTC app. That would immensely help with the team's preparation for the upcoming qualifiers.

[cmacfarl]

@rpallikonda Please check the email that you used to register with FIRST.

[rpallikonda]

Thank you Craig @cmacfarl , I received the invite. I will upload my sample and make progress on the training.
Really appreciate your help in unblocking the team.

[lizlooney]

It's the eval job that's failing to be created. It's using GPU for the eval job. (The eval job is created before the training job so that it is ready to run the evaluation on the first checkpoint produced by the training job, which may happen quickly if the training job is running on TPU.)

[cmacfarl]

Oh right. Thanks for that clarification.

[lizlooney]

The screenshot of the IAM shows that the App Engine default service account has the Editor role. That's the same role that I have on my project. However, I think your Editor role might have different permissions than my Editor role.

Does your Editor role have the roles ml.jobs.create and ml.jobs.get? I think this is the URL to find out...
https://console.cloud.google.com/iam-admin/roles/details/roles%3Ceditor?project=

Also, can you just try it again? I wonder if it is a transitory issue with getting access to a GPU and the error message is not quite accurate.

[rpallikonda]

@lizlooney The Editor role does have the ml.jobs.create and ml.jobs.get permission

s

[rpallikonda]

Also, can you just try it again? I wonder if it is a transitory issue with getting access to a GPU and the error message is not quite accurate.

I tried again and it failed with the same error.