Overview of code loading issues leading to errors or deadlocks with long running http(s) requests

[jrafanie]

Loading code can be a constant problem 🤣

Diagnostics

• This PR Added the logging of long running http(s) requests
• For debugging rails autoload deadlocks, you can use the DebugLocks middleware and hitting via /rails/locks to see what threads are holding locks or waiting on locks.

UI deadlocks

• We now eager load api collection and descendant classes, similar to how we do in the API, see below.

API Deadlocks

• This PR found that loading of code in threads via constantize could cause deadlocks. It resolved this by wrapping API loading of code with ActiveSupport::Dependencies.interlock.permit_concurrent_loads. This issue led to the diagnostic logging of long running requests so we can try to isolate concurrent loadings of code vs. just slow requests.

• This PR found that the prior PR's permit_concurrent_loads is too permissive and we had to use ActiveSupport::Dependencies.interlock.loading when loading code in threads.

• This PR prevents various LoadErrors or possible deadlocks by wrapping autloaded models with interlock loading.

• This PR eager loads all api collections and descendants to avoid deadlocks.

Dalli/memcached deadlocks

• memcached session storage with dalli also uses a lock and needed to ensure code loading while within the critical section of memcached would properly load code safely using permit_concurrent_loads.

Automate deadlocks / errors

• Allow constants to load from DRb threads by using permit_concurrent_loads PR 1 PR 2

• [Automate service resolution const_missing recursively, stack level too deep in dev mode] (Guard against hitting const_missing from within const_missing manageiq-automation_engine#415). It's still not dev mode reloading safe and has load issues as seen above and in this issue.

• More service issues in dev mode, reload! not working

Circular dependency / load ordering issues

• Let Chargeback finish loading before loading ChargebackVm
• Sort glob results of vmdb_extensions to ensure they are loaded alphabetically

[jrafanie]

New possible issue here: ManageIQ/manageiq-ui-classic#7329

[Fryguy]

A lot of this might go away with Rails 6

[Fryguy]

@jrafanie Can you also update the OP in the Diagnostics section with the PR that dumps the threads to see if there are locked puma threads and who is locking who?

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[miq-bot]

This issue has been automatically closed because it has not been updated for at least 3 months.

Feel free to reopen this issue if this issue is still valid.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.