Automate Methods hang after rails 5 upgrade

[mkanoor]

After upgrading to rails 5 automate methods hang in dev and prod environment but works in test env.

[mansam]

I noticed this problem yesterday as well. It seems that it reaches the point where a subprocess is spawned that never wakes up from sleep for whatever reason, causing everything else to hang.

https://github.com/ManageIQ/manageiq/blob/master/lib/miq_automation_engine/engine/miq_ae_method.rb#L270

[chessbyte]

/cc @matthewd @Fryguy

[mkanoor]

This seems to happen because of MiqAePassword being undefined when executing this line of code https://github.com/ManageIQ/manageiq/blob/master/lib/miq_automation_engine/engine/miq_ae_service/miq_ae_service_object_common.rb#L5

if the MiqAePassword is required in lib/miq_automation_engine/engine/miq_ae_service.rb then we don't see this issue.
require_relative '../models/miq_ae_password'
Another twist is this works in TEST but not in DEV or PROD environments.

[jrafanie]

Is it a difference with autoloading of files in test vs. dev/prod from rails 4 to 5? Did we change the autoload paths?

[mkanoor]

@jrafanie
Not sure.
The autoload is properly set in
https://github.com/ManageIQ/manageiq/blob/master/config/application.rb#L75

Stepping thru the code we locate the correct file to be loaded but hang on this line

Dependencies.load_interlock
https://github.com/rails/rails/blob/v5.0.0.beta2/activesupport/lib/active_support/dependencies.rb#L369

https://github.com/rails/rails/blob/v5.0.0.beta2/activesupport/lib/active_support/dependencies/interlock.rb#L11
#<ActiveSupport::Concurrency::ShareLock:0x007faad7d22920 @mon_owner=nil, @mon_count=0, @mon_mutex=#<Mutex:0x007faad7d228d0>, @cv=#<MonitorMixin::ConditionVariable:0x007faad7d228a8 @monitor=#<ActiveSupport::Concurrency::ShareLock:0x007faad7d22920 ...>, @cond=#<Thread::ConditionVariable:0x007faad7d22880>>, @sharing={#<Thread:0x007faac987b808@/Users/xxxxx/.gem/ruby/2.2.2/gems/puma-2.16.0/lib/puma/thread_pool.rb:61 sleep>=>1}, @waiting={}, @exclusive_thread=nil, @exclusive_depth=0>

[jrafanie]

It's a good thing we have the right person for the job @matthewd. 🙇
I've always wanted to extract the run ruby method thing to be runnable from console/rails runner so you don't have to figure up the UI and do a bunch of steps to test it.

[matthewd]

Ohhhhh, I see.

The thing that invokes the drb needs to wrap the invocation in https://github.com/rails/rails/blob/master/activesupport/lib/active_support/dependencies/interlock.rb#L46 -- thereby promising it's not going to run any code that might hit an autoloadable constant inside the block / while it's waiting.