Threads? In Ruby?
Threading in Ruby is something of a sore subject. On the one hand, threads in Ruby are super-easy to work with. If you want to execute something in a thread, you just create a new Thread object, handing it a block containing the code you want to execute:
Thread.new do STDERR.puts "Hello!" # runs in a new thread end
The problem is that people who come from languages like Java often are surprised to hear that although Ruby threads are full-fledged system threads, they also have a global interpreter lock (GIL), which prevents more than one thread from executing at a time. This means that if you spawn 20 threads, you will indeed have 20 threads, but the GIL acts as a big mutex, ensuing that no more than one thread is executing simultaneously. Thread execution typically switches for I/O, and given that nearly every program uses I/O on a regular basis, this almost ensures that each thread will be given a chance to execute.
I should note that Ruby isn't the only language with these issues. Python also has a GIL, and Guido van Rossum, Python's creator, has indicated that although he certainly wants Python to support threading, he personally prefers the ease and security of processes. Because processes don't share state, they are less prone to difficult-to-debug problems, without sacrificing too much in execution speed.
Sidekiq is threaded, but it uses a different model for threads than most Rubyists are used to. It uses Celluloid, an "actor-based" threading system that packages the threads inside objects, avoiding most or all of the issues associated with threads. Moreover, Celluloid expects to run in JRuby or Rubinius, two alternative Ruby implementations, which have true threading and lack the GIL. Celluloid-based applications, such as Sidekiq, will work just fine under the standard Ruby interpreter, known as the MRI, but you won't enjoy all of the speed or threading benefits.
Now, let's see how this overview of a delayed job can be implemented in Sidekiq. First and foremost, you'll need to install the Redis NoSQL store. Redis is available from a variety of sources; I was able to install it on my Ubuntu-based machine with:
apt-get install redis # check this
Once Redis is installed, you'll want to install the "sidekiq" gem. Again, it'll give you the best functionality if you run it under JRuby or Rubinius, but you can run it under the standard Ruby interpreter as well. Just realize that the threads will give you non-optimal performance. You can install the gem with:
sudo gem install sidekiq -V
If you're running the Ruby Version Manager (RVM), as I do, you don't want to install the gem as root. Instead, you should just type:
gem install sidekiq -V
(I always like to use the
-V flag, so I can see the details of
the gem as it is installed.)
You can use Sidekiq in any Ruby application. However, most of my work is in Rails, and I imagine you're going to want to use it in Rails, Sinatra or a similar Web application. Thus, let's create a simple Rails application so you can try it:
rails new appointments
Within the new "appointments" directory, you'll then create an "appointment" resource with scaffolding—a combination of model, controller and view that can get you going quickly:
rails g scaffold appointment name:text ↪meeting_at:timestamp notes:text
Once that is done, you have to run the migration, creating the appropriate "appointments" table in your database. In this case, because you didn't specify a database, you'll use SQLite, which is good enough for this toy example.
Now you can fire up your application (
rails s) and go to
/appointments. From that URL, you can create, view, edit and delete
appointments. However, the point is not to create appointments, but rather
delay the execution of something having to do with them. Let's do
something extremely simple, such as sending e-mail:
rails g mailer notifications
Inside app/mailers/notifications.rb, add the following method:
def appointment_info(person, appointment) @person = person @appointment = appointment mail(to:person.email, subject:"Appointment update") end end
And, inside app/views/notifications/appointment_info.html.erb, write the following:
<p>Hello! You have an appointment with <%= @person %> at <%= @appointment.meeting_at %>.</p>
Finally, let's tie it all together, sending your notification, from within your AppointmentWorker class. There's no rule for where the file defining such a class needs to be put, but it seems increasingly standard to have it in app/workers, in part because files under app are all loaded when Rails starts up:
class AppointmentWorker include Sidekiq::Worker def perform(appointment) Notifications.deliver_appointment_info(appointment) end end
Notice several things here. First, the class doesn't inherit from anything special. Sidekiq doesn't use inheritance, but rather has you include a module—a class without instances, in Ruby—whose methods then are available to instances of your class. This is how the perform_async method is defined on you class. Through a little bit of magic, importing a module can define both class and instance methods.
Now all you have to do is change your controller, such that after you create a report, you also send a notification:
Notice that you're not passing the ID of the appointment, but the appointment object itself! Sidekiq uses Ruby's built-in serialization facility to store nearly any sort of object, not just numeric IDs. The object and method call are stored in Redis, until they are retrieved by a Sidekiq process.
Indeed, that's the final part of Sidekiq that you need to get in place: the back-end process that will look through the delayed jobs, running each one in turn as it gets to them. Fortunately, running that is as easy as:
bundle exec sidekiq
Yup, that's all you need to do. True, there are some options you can set, but generally speaking, this starts up a Sidekiq server that looks at the current queue (as stored in Redis), grabs a job off the queue and processes it. You can configure Sidekiq to run with a timeout or with a specified number of retries, and you even can say how many concurrent workers (that is, threads) you want to be working simultaneously.
Remember that although these are indeed threads, Sidekiq (via Celluloid) ensures that they don't have any state in common. Of course, you need to be sure that your worker methods are thread-safe, such that even if a worker gets through 90% of its job and is then halted, it'll be able to restart without any penalties or errors. Thus, your processes must be transactional, just as you would expect from a database query.
There are other ways to schedule Sidekiq jobs, besides defining methods within a module, as in the above example. If there's an existing method that you want to run as a background process, just insert the "delay" method before the actual method call. That is:
If you are using Rails and the built-in ActiveSupport module for easy time descriptions, you even can do something like this:
Sidekiq has become quite popular in the Ruby community since it was released, in no small part because of its high performance, easy installation and ease of use. It also works with commercial hosting services, such as Heroku, assuming that you first install a Redis instance.
Working with delayed jobs changes your perspective of the Web somewhat—you realize that not everything needs to take place immediately. Rather, you can delay certain jobs, putting them in the background, allowing your Web server to respond to users faster than otherwise would be the case. And, when speed is such a crucial element of Web application success, prudent use of Sidekiq likely will make a big difference.
The Sidekiq home page is at http://sidekiq.org. Although Sidekiq.org does point to a commercial version, the basic version is still free and open source, with the source code available on GitHub at http://mperham.github.com/sidekiq, including a Wiki containing a great deal of useful information.
Mike Perham, the author of Sidekiq, describes the actor-based model in a blog post: http://blog.carbonfive.com/2011/04/19/concurrency-with-actors.
Finally, given that Sidekiq uses Redis, you likely will want to read more about this high-performance NoSQL database, at http://redis.io.
Reuven M. Lerner, Linux Journal Senior Columnist, a longtime Web developer, consultant and trainer, is completing his PhD in learning sciences at Northwestern University.
Free DevOps eBooks, Videos, and more!
Regardless of where you are in your DevOps process, Linux Journal can help!
We offer here the DEFINITIVE DevOps for Dummies, a mobile Application Development Primer, and advice & help from the expert sources like:
- Linux Journal
Web Development News
- New Products
- New Products
- Integrating Trac, Jenkins and Cobbler—Customizing Linux Operating Systems for Organizational Needs
- Tech Tip: Really Simple HTTP Server with Python
- RSS Feeds
- Dialog: An Introductory Tutorial
- Non-Linux FOSS: Remember Burning ISOs?
- Returning Values from Bash Functions
- EdgeRouter Lite