Improve back-off client

[lavigne958]

Catch usage limit errors from the Drive API. it returns the HTTP code 403 (access denied) on usage limits and access denied. Find the difference in both case and retry when it's usage limits exceeded.

Improve back-off client code.

Add pretty print for the APIerror Exception.

closes #1268
closes #1326

[lavigne958]

Trying to reach the API limit on the Drive API was such a pain 💥 I had to run 600 requests in parallel in order to reach the limit of 12000 requests per 60 seconds....

I believe not many users will have to backoff from this API 🙄

[alifeee]

these changes look good to me. two things:

• the test in Fix BackOffHTTPClient not returning error 403 #1268 needs to be un-skipped, or the issue is not closed
• we should add a test with this new functionality.

[lavigne958]

these changes look good to me. two things:

• the test in Fix BackOffHTTPClient not returning error 403 #1268 needs to be un-skipped, or the issue is not closed
• we should add a test with this new functionality.

I enabled the previously skipped test.

I am not sure how to test this new code, we have a test that tries to open a forbidden spreadsheet.
We can't make a test that reaches the limits of the quotas by sending many requests: as this will take forever to record the first time and everytime we update the cassettes. last time I ran this code I could not reach the limit of quota due to the backoff client 😵‍💫

[alifeee]

Can we mock the requests object for just this one test? Instead of using cassettes, we could mock requests to return 419 (or the right HTTP #), and then test that the client works as expected (likely by setting the timeout very low so it only retries once or twice)

[lavigne958]

Can we mock the requests object for just this one test? Instead of using cassettes, we could mock requests to return 419 (or the right HTTP #), and then test that the client works as expected (likely by setting the timeout very low so it only retries once or twice)

I ran some tests and I can't figure it out, I can change requests before they are made but they still need to be made, I can filter/change responses before they are returned to us but requests still need to be made with a valid response.

I can't find a way to filter that request specifically and change it's return code, in a some easy/simple ways.

it looks like a lot of work for a automatic test.

is it ok for you if we keep it like this for now ?

[alifeee]

is it ok for you if we keep it like this for now ?

yes :) I think people will probably not hit the rate limit enough that it is that important...

[lavigne958]

is it ok for you if we keep it like this for now ?

yes :) I think people will probably not hit the rate limit enough that it is that important...

thank you, I agree, I tried it myself... 😆 never reached it