Sidekiq

Threads? In Ruby?

Threading in Ruby is something of a sore subject. On the one hand, threads in Ruby are super-easy to work with. If you want to execute something in a thread, you just create a new Thread object, handing it a block containing the code you want to execute:


Thread.new do
    STDERR.puts "Hello!"  # runs in a new thread
end

The problem is that people who come from languages like Java often are surprised to hear that although Ruby threads are full-fledged system threads, they also have a global interpreter lock (GIL), which prevents more than one thread from executing at a time. This means that if you spawn 20 threads, you will indeed have 20 threads, but the GIL acts as a big mutex, ensuing that no more than one thread is executing simultaneously. Thread execution typically switches for I/O, and given that nearly every program uses I/O on a regular basis, this almost ensures that each thread will be given a chance to execute.

I should note that Ruby isn't the only language with these issues. Python also has a GIL, and Guido van Rossum, Python's creator, has indicated that although he certainly wants Python to support threading, he personally prefers the ease and security of processes. Because processes don't share state, they are less prone to difficult-to-debug problems, without sacrificing too much in execution speed.

Sidekiq is threaded, but it uses a different model for threads than most Rubyists are used to. It uses Celluloid, an "actor-based" threading system that packages the threads inside objects, avoiding most or all of the issues associated with threads. Moreover, Celluloid expects to run in JRuby or Rubinius, two alternative Ruby implementations, which have true threading and lack the GIL. Celluloid-based applications, such as Sidekiq, will work just fine under the standard Ruby interpreter, known as the MRI, but you won't enjoy all of the speed or threading benefits.

Using Sidekiq

Now, let's see how this overview of a delayed job can be implemented in Sidekiq. First and foremost, you'll need to install the Redis NoSQL store. Redis is available from a variety of sources; I was able to install it on my Ubuntu-based machine with:


apt-get install redis   # check this

Once Redis is installed, you'll want to install the "sidekiq" gem. Again, it'll give you the best functionality if you run it under JRuby or Rubinius, but you can run it under the standard Ruby interpreter as well. Just realize that the threads will give you non-optimal performance. You can install the gem with:


sudo gem install sidekiq -V

If you're running the Ruby Version Manager (RVM), as I do, you don't want to install the gem as root. Instead, you should just type:


gem install sidekiq -V

(I always like to use the -V flag, so I can see the details of the gem as it is installed.)

You can use Sidekiq in any Ruby application. However, most of my work is in Rails, and I imagine you're going to want to use it in Rails, Sinatra or a similar Web application. Thus, let's create a simple Rails application so you can try it:


rails new appointments

Within the new "appointments" directory, you'll then create an "appointment" resource with scaffolding—a combination of model, controller and view that can get you going quickly:


rails g scaffold appointment name:text 
 ↪meeting_at:timestamp notes:text

Once that is done, you have to run the migration, creating the appropriate "appointments" table in your database. In this case, because you didn't specify a database, you'll use SQLite, which is good enough for this toy example.

Now you can fire up your application (rails s) and go to /appointments. From that URL, you can create, view, edit and delete appointments. However, the point is not to create appointments, but rather delay the execution of something having to do with them. Let's do something extremely simple, such as sending e-mail:


rails g mailer notifications

Inside app/mailers/notifications.rb, add the following method:


def appointment_info(person, appointment)
    @person = person
    @appointment = appointment
    mail(to:person.email, subject:"Appointment update")
    end
end

And, inside app/views/notifications/appointment_info.html.erb, write the following:


<p>Hello! You have an appointment with <%= @person %>
at <%= @appointment.meeting_at %>.</p>

Finally, let's tie it all together, sending your notification, from within your AppointmentWorker class. There's no rule for where the file defining such a class needs to be put, but it seems increasingly standard to have it in app/workers, in part because files under app are all loaded when Rails starts up:


class AppointmentWorker
  include Sidekiq::Worker

  def perform(appointment)
    Notifications.deliver_appointment_info(appointment)
  end
end

Notice several things here. First, the class doesn't inherit from anything special. Sidekiq doesn't use inheritance, but rather has you include a module—a class without instances, in Ruby—whose methods then are available to instances of your class. This is how the perform_async method is defined on you class. Through a little bit of magic, importing a module can define both class and instance methods.

Now all you have to do is change your controller, such that after you create a report, you also send a notification:


AppointmentWorker.perform_async(@appointment)

Notice that you're not passing the ID of the appointment, but the appointment object itself! Sidekiq uses Ruby's built-in serialization facility to store nearly any sort of object, not just numeric IDs. The object and method call are stored in Redis, until they are retrieved by a Sidekiq process.

Indeed, that's the final part of Sidekiq that you need to get in place: the back-end process that will look through the delayed jobs, running each one in turn as it gets to them. Fortunately, running that is as easy as:


bundle exec sidekiq

Yup, that's all you need to do. True, there are some options you can set, but generally speaking, this starts up a Sidekiq server that looks at the current queue (as stored in Redis), grabs a job off the queue and processes it. You can configure Sidekiq to run with a timeout or with a specified number of retries, and you even can say how many concurrent workers (that is, threads) you want to be working simultaneously.

Remember that although these are indeed threads, Sidekiq (via Celluloid) ensures that they don't have any state in common. Of course, you need to be sure that your worker methods are thread-safe, such that even if a worker gets through 90% of its job and is then halted, it'll be able to restart without any penalties or errors. Thus, your processes must be transactional, just as you would expect from a database query.

There are other ways to schedule Sidekiq jobs, besides defining methods within a module, as in the above example. If there's an existing method that you want to run as a background process, just insert the "delay" method before the actual method call. That is:


my_object.delay.do_something_big

If you are using Rails and the built-in ActiveSupport module for easy time descriptions, you even can do something like this:


my_object.delay_for(5.days).do_something_big

Conclusion

Sidekiq has become quite popular in the Ruby community since it was released, in no small part because of its high performance, easy installation and ease of use. It also works with commercial hosting services, such as Heroku, assuming that you first install a Redis instance.

Working with delayed jobs changes your perspective of the Web somewhat—you realize that not everything needs to take place immediately. Rather, you can delay certain jobs, putting them in the background, allowing your Web server to respond to users faster than otherwise would be the case. And, when speed is such a crucial element of Web application success, prudent use of Sidekiq likely will make a big difference.

Resources

The Sidekiq home page is at http://sidekiq.org. Although Sidekiq.org does point to a commercial version, the basic version is still free and open source, with the source code available on GitHub at http://mperham.github.com/sidekiq, including a Wiki containing a great deal of useful information.

Mike Perham, the author of Sidekiq, describes the actor-based model in a blog post: http://blog.carbonfive.com/2011/04/19/concurrency-with-actors.

Finally, given that Sidekiq uses Redis, you likely will want to read more about this high-performance NoSQL database, at http://redis.io.

______________________

Reuven M. Lerner, Linux Journal Senior Columnist, a longtime Web developer, consultant and trainer, is completing his PhD in learning sciences at Northwestern University.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Woah that blog is usually

Anonymous's picture

Woah that blog is usually excellent i really like reading ones articles. Stay in the good paintings! You already know, a lot of individuals are searching around due to this info, you can help these individuals enormously. http://shortnatural-hairstyles.blogspot.ro

Page loading

Taylor's picture

Nowadays, the chance of users leaving a web page are very high if the page is loading slowly. That's why reducing the loading times in a web application is very important. Sidekiq is a must have for dividing jobs due to its high performance.

Good Perspective

Kappy's picture

Ah, there is a great perspective followed by an amazing piece of code.

Thanks for the

Lauri Nevala's picture

Thanks for the introduction!

As a side note. I would rather pass the id of the user to the worker instead of the user object. Here are more details about this issue:
https://github.com/mperham/sidekiq/wiki/Best-Practices#1-make-your-jobs-...

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix