[BUG]: PowerStore Replication - Delete RG request hangs

[santhoshatdell]

Bug Description

When the source RG is requested to be deleted on PowerStore, it doesn't finish and the test eventually times out.

Reason: Request to remove the associated protection policy before deleting the volume group doesn't remove the policy. Please see the logs. It was a regression introduced in gopowerstore module when there were volume group updates.

Workaround: User to manually disassociate the protection policy from the source side volume group when the RG deletion call hangs.

Logs

VolumeGroup PATCH request only has description in it. It used to send empty string for protectionPolicyId.

{"level":"debug","msg":"REQUEST: PATCH /api/rest/volume_group/5ac8d0ab-9xxx-b982 HTTP/1.1 Host: 10.230.24.33 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ****** {"description":""}","time":"2023-08-01T21:20:56.873621998Z"}

Driver log:
{"GlobalID":"PS123456789","ProtectedStorageGroup":"5ac8d0ab-9xxx-b982","level":"info","msg":"Deleting storage protection group","time":"2023-08-01T21:20:56.83085518Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/volume_group/5ac8d0ab-9xxx-b982?select=%2A%2Cvolumes%28%2A%29%2Cprotection_policy%28%2A%29%2Cprotection_data%2Clocation_history%2Cmigration_session%28%2A%29%2Cvolume.volume_group_membership%28id%2Cname%2Cprotection_policy_id%2Cstate%2Cprotection_data%29 HTTP/1.1 Host: 10.230.24.33 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ****** ","time":"2023-08-01T21:20:56.831004384Z"}
{"level":"debug","msg":"acquire a lock","time":"2023-08-01T21:20:56.831024612Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 1625 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff {"id":"5ac8d0ab-9xxx-b982","name":"vg-xxx","description":null,"creation_timestamp":"2023-08-01T21:04:03.272887+00:00","is_protectable":true,"protection_policy_id":"177e233a-30f0-4393-a5bf-b39d66db1c04","migration_session_id":null,"is_write_order_consistent":false,"placement_rule":"Same_Appliance","type":"Primary","is_replication_destination":false,"protection_data":{"family_id": "5ac8d0ab-9xxx-b982", "is_secure": false, "parent_id": null, "source_id": null, "creator_type": "User", "copy_signature": null, "source_timestamp": "2023-08-01T21:08:32.217048+00:00", "creator_type_l10n": "User", "is_app_consistent": false, "created_by_rule_id": null, "created_by_rule_name": null, "expiration_timestamp": null},"is_importing":false,"location_history":null,"type_l10n":"Primary","protection_data":{"family_id": "5ac8d0ab-9xxx-b982", "is_secure": false, "parent_id": null, "source_id": null, "creator_type": "User", "copy_signature": null, "source_timestamp": "2023-08-01T21:08:32.217048+00:00", "creator_type_l10n": "User", "is_app_consistent": false, "created_by_rule_id": null, "created_by_rule_name": null, "expiration_timestamp": null},"location_history":null,"volumes":[],"protection_policy":{"id":"177e233a-30f0-4393-a5bf-b39d66db1c04","name":"pp-vg-xxx","description":null,"type":"Protection","managed_by":"User","managed_by_id":null,"is_read_only":false,"is_replica":false,"type_l10n":"Protection","managed_by_l10n":"Managed by the end user"},"migration_session":null,"volume":[]}\n","time":"2023-08-01T21:20:56.873380591Z"}
{"level":"debug","msg":"release a lock","time":"2023-08-01T21:20:56.873541471Z"}
{"level":"debug","msg":"REQUEST: PATCH /api/rest/volume_group/5ac8d0ab-9xxx-b982 HTTP/1.1 Host: 10.230.24.33 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ****** {"description":""}","time":"2023-08-01T21:20:56.873621998Z"}
{"level":"debug","msg":"acquire a lock","time":"2023-08-01T21:20:56.873639874Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 204 No Content Content-Length: 0 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff \n","time":"2023-08-01T21:20:56.951735617Z"}
{"level":"debug","msg":"release a lock","time":"2023-08-01T21:20:56.951803793Z"}
{"level":"debug","msg":"REQUEST: DELETE /api/rest/volume_group/5ac8d0ab-9xxx-b982 HTTP/1.1 Host: 10.230.24.33 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ****** ","time":"2023-08-01T21:20:56.95187102Z"}
{"level":"debug","msg":"acquire a lock","time":"2023-08-01T21:20:56.951895818Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 422 Unprocessable Entity Transfer-Encoding: chunked Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 X-Content-Type-Options: nosniff 165 {"messages":[{"code":"0xE0A070040002","severity":"Error","message_l10n":"Remove the protection policy from the volume group vg-xxx (id: 5ac8d0ab-9xxx-b982) before deleting the volume group.","arguments":["5ac8d0ab-9xxx-b982","vg-xxx"]}]} 0 \n","time":"2023-08-01T21:20:57.020292284Z"}
{"level":"debug","msg":"release a lock","time":"2023-08-01T21:20:57.020362803Z"}
{"level":"info","msg":"/replication.v1.Replication/DeleteStorageProtectionGroup: REP 0465: rpc error: code = Internal desc = Error: Remove the protection policy from the volume group vg-xxx (id: 5ac8d0ab-9xxx-b982) before deleting the volume group.: Unable to delete Volume Group","time":"2023-08-01T21:20:57.020407155Z"}

Screenshots

No response

Additional Environment Information

No response

Steps to Reproduce

• Create PV with replication enabled SC
• Delete PV
• Delete RG

Expected Behavior

RG deletion to succeed.

CSM Driver(s)

CSI Driver for PowerStore v2.7.0
gopowerstore v1.12.0

Installation Type

No response

Container Storage Modules Enabled

Replication v1.5.0

Container Orchestrator

Kubernetes 1.27

Operating System

RHEL 8.7

[csmbot]

@santhoshatdell: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and assign an appropriate priority label.

---

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at container.storage.modules@dell.com.