Lose access to ActiveRecord in daemon mode

[tonyjiang]

I have a background job in which I need to access ActiveRecord models. If I start it without '-d', things work fine - I can access the database through AR models; but once I put '-d' in the command, the script doesn't work anymore.

I wonder if anybody has seen this before?

[colinsurprenant]

do you have any errors/exceptions? which Ruby & version are you using? could you gist a minimal example which reproduces the problem so I can take a look?

[tonyjiang]

I am using JRuby 1.7.1. Here is my script in its entirety:

require 'raad'
require 'rufus-scheduler'

class Scheduler
  def start
    @scheduler ||= Rufus::Scheduler.start_new

    @scheduler.every '10s' do
      File.open("tmp/foo", 'a') do |file|
        Report.all.each do |report|
          file.puts "    " + report.name.to_s + " === " + report.type.to_s + "==== #{Time.now}"
        end
      end
    end      

    @scheduler.join
  end

  def stop
  end
end

It works fine if I run it like this:
bundle exec rails runner lib/scheduler/scheduler.rb -P tmp/scheduler.pid -l tmp/scheduler.log start

We know it works because file 'tmp/foo' is populated with data from database. But if I run the script with '-d' like so:
bundle exec rails runner lib/scheduler/scheduler.rb -d -P tmp/scheduler.pid -l tmp/scheduler.log start

File 'tmp/foo' remains empty and there is no feedback - error/exception/backtrace.

To prove that daemon mode works without ActiveRecord ('Report' in my script), I changed it to:

require 'raad'
require 'rufus-scheduler'

class Scheduler
  def start
    @scheduler ||= Rufus::Scheduler.start_new

    @scheduler.every '10s' do
      File.open("tmp/foo", 'a') do |file|
        # Report.all.each do |report|
        #   file.puts "    " + report.name.to_s + " === " + report.type.to_s + "==== #{Time.now}"
        # end
        file.puts "output from scheduler ... #{Time.now}"
      end
    end      

    @scheduler.join
  end

  def stop
  end
end

It indeed works fine - 'tmp/foo' is populated with data.

[colinsurprenant]

Right. I haven't had time to test your code but the problem I see is that with JRuby, the daemonizing process involves the respawning of the process (cannot fork on the JVM). Since the process itself is launched within the rails context using rails runner it probably does not capture this and respawn only the daemon code without the rails context. I will have to trace to see if this indeed the problem and how/if we can fix.

I think it should work better using MRI Ruby, because the daemonizing process is done using a double fork which "keeps" the process context. If this is an acceptable temporary solution to have this daemon run in MRI Ruby maybe you could try it while we figure out how to make that work in JRuby.

[tonyjiang]

I think you are spot on with Rails runner not being captured by Raad since the process is launched within Rails.

I thought about running the script through a Rake task like so:

  task :start => [:environment] do
      load "lib/scheduler.rb", "-d start"
  end

But can't find a way to pass the parameters (the above task doesn't work). Not sure if there is a way to do it.

MRI definitely works but unfortunately it's not an option for me.

[colinsurprenant]

Maybe a quick & easy alternative would be to include activerecord from within your daemon and set it up manually using something like

ActiveRecord::Base.establish_connection(
  :adapter => '...',
  :database => '...'
)

then just start the daemon without the rails runner. The disadvantage is that you will have to manage your db config also for the daemon instead of relying on your Rails app config. The advantage is that you won't need to load the complete Rails environment just to run this daemon.

Let me know if that works.

[tonyjiang]

Because I am working with a substantial Rails app, it's very difficult to set up things to just use AR. For example, this is how far I went before giving up:

require 'rubygems'
require 'rufus-scheduler'
require 'active_record'
require 'activerecord-jdbc-adapter'
require 'jdbc-mssql-azure'
require 'active_record/connection_adapters/jdbcmssql_adapter'

class AvocadoScheduler
  def start
    @scheduler ||= Rufus::Scheduler.start_new
    ActiveRecord::Base.establish_connection(
      :adapter => 'jdbcmssql',
      :database => '...',
      :username => '...',
      :password => '...',
      :driver => '...',
      :url => '...'
    )

    require 'acts-as-taggable-on'
    require File.expand_path('../../../app/models/report', __FILE__)
    require File.expand_path('../../../app/models/reports/foo_report', __FILE__)
    require File.expand_path('../../../app/models/reports/bar_report', __FILE__)
    require File.expand_path('../../../app/models/reports/baz_report', __FILE__)
    # .......

It's quite hacky, messy, and error-prone. It doesn't look like a viable solution.

[colinsurprenant]

I agree this is not the best solution. I will check to see if it is possible to make this work within rails runner but clearly I did not have Rails in mind as a usage pattern with creating raad :P

Otherwise, did you check background job processors tailored for Rails like Resque https://github.com/defunkt/resque or Sidekiq https://github.com/mperham/sidekiq ? These might just be better solutions for your context?

Let me know.

[tonyjiang]

Those two tools serve a different purpose: I want to run a scheduler at a fixed interval (say 1m);I I don't need to queue anything. I want to daemonize the scheduler so it's not tied to Rails (for example if the web server crashes, the scheduler will not be affected). I am also reluctant to introduce an extra component (Redis) into my application.

[tonyjiang]

It's likely that my team will use Ubuntu's upstart (which I know precious little about) to manage the daemonizing process. So we probably won't use Raad in production.

Thank you for all the help, Colin.

[colinsurprenant]

Make sense. I mean, there's plenty of options here to make this work, starting with a simple cron job. You should take a look at the whenever gem https://github.com/javan/whenever and maybe clockwork https://github.com/adamwiggins/clockwork .

Let me know what will end up working for you. On my side I will definitely take a look to see if raad ca be used in the context of rails runner with JRuby.