Migrating From Sidekiq To Solid Queue: Simpler and Cheaper

Table of Contents

• Sidekiq Is Great!
• Why Make The Switch To Solid Queue?
• 😄 Reduced Redis Usage
• 😄 Built-in Cron Runner
• 😄 Less Processes → Cost Savings
• 😔 “Mission Control” Can Be Improved

Sidekiq Is Great!

Sidekiq has been the go-to choice for Ruby on Rails applications for years. And for good reason — it’s packed with incredible features.

Huge shoutout to the team for open-sourcing such a powerful tool!

Why Make The Switch To Solid Queue?

Late last year, DHH unveiled Solid Queue - a new gem to leverage your existing database to handle background jobs. And the results speak for themselves: 37signals is processing about 5.6 million jobs daily with it.

Solid Queue, the new DB-based backend for Active Job in Rails, has made its premiere! We're running millions of jobs through it every day, and replaced a plethora of Resque gems to get the introspection and features we need. Check it out: https://t.co/lxqrl2vjoK

— DHH (@dhh) December 18, 2023

Simplifying our codebase like that is incredibly appealing, so naturally, I had to dive deeper into it.

The setup was a breeze, and we wrapped up the entire migration, including testing, in a single day.

Here’s my take on Solid Queue compared to Sidekiq—what I love and what I don’t.

😄 Reduced Redis Usage

Sure, Redis wasn’t breaking the bank on our cloud bill, but it was pulling double duty as our job scheduler and cache store.

By ditching Redis for Sidekiq, we get two wins:

1. A leaner, cheaper Redis setup
2. More room for our cache, which means fewer evictions, more cache hits, and snappier responses.

😄 Built-in Cron Runner

Before, when using Sidekiq, we had to scramble for a job scheduler since Sidekiq doesn’t come with a built-in cron feature.

Take this for example: we had to install the Whenever gem to handle job scheduling. And because we’re deploying via a Dockerfile to Fly.io, we also needed to install cron and set up Supercronic as our job runner.

But with Solid Queue, all that hassle is gone! Now, we can manage everything right in the solid_queue.yml file. It’s streamlined, and it just works.

😄 Less Processes → Cost Savings

Previously, using Sidekiq meant spinning up a separate machine for each process handling different queues. This was inefficient, as these machines often sat idle, wasting resources. For instance, we had to deploy four processes—one for cron jobs and three for Sidekiq.

# We are using Flu.io, thus defining processes within fly.toml
# Reference: https://fly.io/docs/reference/configuration/
[processes]
  cron = "supercronic ./crontab"
  sidekiq_1 = "bundle exec sidekiq -c 100 -q logging -q default -q llm"
  sidekiq_2 = "bundle exec sidekiq -c 10 -q mailers -q scrapers"
  sidekiq_3 = "bundle exec sidekiq -c 1 -q refresh_salary_views"

With Solid Queue, all workers run on the same machine. Plus, with Cron now integrated, there’s no need for a separate machine for cron jobs. We’ve reduced our hardware requirements from four machines to just one—an efficient and cost-effective solution!

# solid_queue.yml
workers:
  - queues: [logging, default, llm]
    threads: 100
  - queues: [mailers, scrapers]
    threads: 10
  - queues: [refresh_salary_views]
    threads: 1

😔 “Mission Control” Can Be Improved

This isn’t a deal breaker by any means, but my greedy desires show up here.

Unlike Sidekiq, SolidQueue doesn’t come with a built-in dashboard. You’ll need to install a separate gem called Mission Control for that. No biggie.

And for the most part, Mission Control does a solid job at mimicking Sidekiq’s dashboard.

But it’d be fantastic if it also included charts to quickly visualize the number of jobs processed and the average time taken for each type. That kind of insight can be pretty useful for debugging and optimization.