Sidekiq

Threads? In Ruby?

Threading in Ruby is something of a sore subject. On the one hand, threads in Ruby are super-easy to work with. If you want to execute something in a thread, you just create a new Thread object, handing it a block containing the code you want to execute:


Thread.new do
    STDERR.puts "Hello!"  # runs in a new thread
end

The problem is that people who come from languages like Java often are surprised to hear that although Ruby threads are full-fledged system threads, they also have a global interpreter lock (GIL), which prevents more than one thread from executing at a time. This means that if you spawn 20 threads, you will indeed have 20 threads, but the GIL acts as a big mutex, ensuing that no more than one thread is executing simultaneously. Thread execution typically switches for I/O, and given that nearly every program uses I/O on a regular basis, this almost ensures that each thread will be given a chance to execute.

I should note that Ruby isn't the only language with these issues. Python also has a GIL, and Guido van Rossum, Python's creator, has indicated that although he certainly wants Python to support threading, he personally prefers the ease and security of processes. Because processes don't share state, they are less prone to difficult-to-debug problems, without sacrificing too much in execution speed.

Sidekiq is threaded, but it uses a different model for threads than most Rubyists are used to. It uses Celluloid, an "actor-based" threading system that packages the threads inside objects, avoiding most or all of the issues associated with threads. Moreover, Celluloid expects to run in JRuby or Rubinius, two alternative Ruby implementations, which have true threading and lack the GIL. Celluloid-based applications, such as Sidekiq, will work just fine under the standard Ruby interpreter, known as the MRI, but you won't enjoy all of the speed or threading benefits.

Using Sidekiq

Now, let's see how this overview of a delayed job can be implemented in Sidekiq. First and foremost, you'll need to install the Redis NoSQL store. Redis is available from a variety of sources; I was able to install it on my Ubuntu-based machine with:


apt-get install redis   # check this

Once Redis is installed, you'll want to install the "sidekiq" gem. Again, it'll give you the best functionality if you run it under JRuby or Rubinius, but you can run it under the standard Ruby interpreter as well. Just realize that the threads will give you non-optimal performance. You can install the gem with:


sudo gem install sidekiq -V

If you're running the Ruby Version Manager (RVM), as I do, you don't want to install the gem as root. Instead, you should just type:


gem install sidekiq -V

(I always like to use the -V flag, so I can see the details of the gem as it is installed.)

You can use Sidekiq in any Ruby application. However, most of my work is in Rails, and I imagine you're going to want to use it in Rails, Sinatra or a similar Web application. Thus, let's create a simple Rails application so you can try it:


rails new appointments

Within the new "appointments" directory, you'll then create an "appointment" resource with scaffolding—a combination of model, controller and view that can get you going quickly:


rails g scaffold appointment name:text 
 ↪meeting_at:timestamp notes:text

Once that is done, you have to run the migration, creating the appropriate "appointments" table in your database. In this case, because you didn't specify a database, you'll use SQLite, which is good enough for this toy example.

Now you can fire up your application (rails s) and go to /appointments. From that URL, you can create, view, edit and delete appointments. However, the point is not to create appointments, but rather delay the execution of something having to do with them. Let's do something extremely simple, such as sending e-mail:


rails g mailer notifications

Inside app/mailers/notifications.rb, add the following method:


def appointment_info(person, appointment)
    @person = person
    @appointment = appointment
    mail(to:person.email, subject:"Appointment update")
    end
end

And, inside app/views/notifications/appointment_info.html.erb, write the following:


<p>Hello! You have an appointment with <%= @person %>
at <%= @appointment.meeting_at %>.</p>

Finally, let's tie it all together, sending your notification, from within your AppointmentWorker class. There's no rule for where the file defining such a class needs to be put, but it seems increasingly standard to have it in app/workers, in part because files under app are all loaded when Rails starts up:


class AppointmentWorker
  include Sidekiq::Worker

  def perform(appointment)
    Notifications.deliver_appointment_info(appointment)
  end
end

Notice several things here. First, the class doesn't inherit from anything special. Sidekiq doesn't use inheritance, but rather has you include a module—a class without instances, in Ruby—whose methods then are available to instances of your class. This is how the perform_async method is defined on you class. Through a little bit of magic, importing a module can define both class and instance methods.

Now all you have to do is change your controller, such that after you create a report, you also send a notification:


AppointmentWorker.perform_async(@appointment)

Notice that you're not passing the ID of the appointment, but the appointment object itself! Sidekiq uses Ruby's built-in serialization facility to store nearly any sort of object, not just numeric IDs. The object and method call are stored in Redis, until they are retrieved by a Sidekiq process.

Indeed, that's the final part of Sidekiq that you need to get in place: the back-end process that will look through the delayed jobs, running each one in turn as it gets to them. Fortunately, running that is as easy as:


bundle exec sidekiq

Yup, that's all you need to do. True, there are some options you can set, but generally speaking, this starts up a Sidekiq server that looks at the current queue (as stored in Redis), grabs a job off the queue and processes it. You can configure Sidekiq to run with a timeout or with a specified number of retries, and you even can say how many concurrent workers (that is, threads) you want to be working simultaneously.

Remember that although these are indeed threads, Sidekiq (via Celluloid) ensures that they don't have any state in common. Of course, you need to be sure that your worker methods are thread-safe, such that even if a worker gets through 90% of its job and is then halted, it'll be able to restart without any penalties or errors. Thus, your processes must be transactional, just as you would expect from a database query.

There are other ways to schedule Sidekiq jobs, besides defining methods within a module, as in the above example. If there's an existing method that you want to run as a background process, just insert the "delay" method before the actual method call. That is:


my_object.delay.do_something_big

If you are using Rails and the built-in ActiveSupport module for easy time descriptions, you even can do something like this:


my_object.delay_for(5.days).do_something_big

Conclusion

Sidekiq has become quite popular in the Ruby community since it was released, in no small part because of its high performance, easy installation and ease of use. It also works with commercial hosting services, such as Heroku, assuming that you first install a Redis instance.

Working with delayed jobs changes your perspective of the Web somewhat—you realize that not everything needs to take place immediately. Rather, you can delay certain jobs, putting them in the background, allowing your Web server to respond to users faster than otherwise would be the case. And, when speed is such a crucial element of Web application success, prudent use of Sidekiq likely will make a big difference.

Resources

The Sidekiq home page is at http://sidekiq.org. Although Sidekiq.org does point to a commercial version, the basic version is still free and open source, with the source code available on GitHub at http://mperham.github.com/sidekiq, including a Wiki containing a great deal of useful information.

Mike Perham, the author of Sidekiq, describes the actor-based model in a blog post: http://blog.carbonfive.com/2011/04/19/concurrency-with-actors.

Finally, given that Sidekiq uses Redis, you likely will want to read more about this high-performance NoSQL database, at http://redis.io.

______________________

Reuven M. Lerner, Linux Journal Senior Columnist, a longtime Web developer, consultant and trainer, is completing his PhD in learning sciences at Northwestern University.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Woah that blog is usually

Anonymous's picture

Woah that blog is usually excellent i really like reading ones articles. Stay in the good paintings! You already know, a lot of individuals are searching around due to this info, you can help these individuals enormously. http://shortnatural-hairstyles.blogspot.ro

Page loading

Taylor's picture

Nowadays, the chance of users leaving a web page are very high if the page is loading slowly. That's why reducing the loading times in a web application is very important. Sidekiq is a must have for dividing jobs due to its high performance.

Good Perspective

Kappy's picture

Ah, there is a great perspective followed by an amazing piece of code.

Thanks for the

Lauri Nevala's picture

Thanks for the introduction!

As a side note. I would rather pass the id of the user to the worker instead of the user object. Here are more details about this issue:
https://github.com/mperham/sidekiq/wiki/Best-Practices#1-make-your-jobs-...

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState