Visual Sidekiq Job Monitoring with Redimo

Sidekiq's Web UI shows you queues, retries, and dead jobs. It's solid for day-to-day monitoring.

But when things go sideways - jobs mysteriously disappearing, memory spiking, workers stuck - you need to look deeper. At the Redis level.

What Sidekiq Stores in Redis

Sidekiq uses a consistent key structure:

sidekiq:queues                    # Set of all queue names
sidekiq:queue:default             # List of jobs in "default" queue
sidekiq:queue:critical            # List of jobs in "critical" queue
sidekiq:retry                     # Sorted set of retry jobs (by next retry time)
sidekiq:dead                      # Sorted set of dead jobs (by death time)
sidekiq:schedule                  # Sorted set of scheduled jobs
sidekiq:processes                 # Set of active Sidekiq processes
sidekiq:<process-id>              # Hash of process info
sidekiq:stat:processed            # Counter of processed jobs
sidekiq:stat:failed               # Counter of failed jobs
sidekiq:stat:processed:YYYY-MM-DD # Daily processed count

Each job in a queue is a JSON blob:

{
  "class": "HardWorker",
  "args": [123, "foo"],
  "retry": 3,
  "queue": "default",
  "jid": "abc123xyz",
  "created_at": 1705312245.123,
  "enqueued_at": 1705312245.456
}

Beyond the Web UI

Sidekiq Web is great for:

• Seeing queue lengths
• Manually retrying jobs
• Viewing recent failures

It's not great for:

• Analyzing job data patterns
• Finding jobs with specific arguments
• Bulk operations on thousands of jobs
• Investigating memory issues
• Comparing queue states across time

Pattern-Based Monitoring

Monitor All Sidekiq Data

Create a Pattern Monitor in Redimo:

Pattern: sidekiq:*

This captures everything Sidekiq-related. The tree view groups by:

• sidekiq:queue:* - All queues
• sidekiq:retry - Retry set
• sidekiq:dead - Dead letter queue
• sidekiq:processes - Active workers
• sidekiq:stat:* - Statistics

Monitor Specific Queues

For focused debugging:

Pattern: sidekiq:queue:critical

See all jobs waiting in the critical queue. Click any to view the full JSON payload - class, arguments, retry count, timestamps.

Common Debugging Scenarios

Scenario 1: Jobs Stuck in Queue

Queue length keeps growing. Workers seem to be running but jobs aren't processing.

Check 1: Are workers connected?

Look at sidekiq:processes. This set contains all active Sidekiq processes.

SMEMBERS sidekiq:processes

Empty? No workers are running. Full of entries from hours ago? Workers died without cleaning up.

Check 2: What's actually in the queue?

Click into sidekiq:queue:default. Look at the job payloads. Are they all the same job class? Same arguments?

Pattern: A batch job that spawned millions of child jobs. Each child is tiny but there are too many.

Check 3: Are workers dying mid-job?

Sidekiq uses brpoplpush for reliable queue processing. Jobs move from the queue to a processing list, then get removed when done.

If workers die mid-job, those jobs vanish (unless you have Sidekiq Pro's reliable fetch).

Scenario 2: Memory Explosion

Redis memory growing fast. Sidekiq seems fine.

Likely culprits:

1. Job arguments too large

Click into queue jobs. Are arguments huge? Storing entire objects instead of IDs?

# Bad - stores entire user object
UserMailer.perform_async(user.to_json)

# Good - stores ID, fetches in worker
UserMailer.perform_async(user.id)

2. Dead jobs accumulating

Check sidekiq:dead size. Default keeps 10,000 dead jobs for 6 months. Each job with its arguments.

# In initializer
Sidekiq.configure_server do |config|
  config.death_handlers << ->(job, ex) do
    # Custom handling - maybe don't keep huge jobs
  end
end

# Or configure limits
Sidekiq.options[:dead_max_jobs] = 1000

3. Retry queue backlog

sidekiq:retry is a sorted set. If jobs keep failing and retrying, this grows.

In Redimo: Filter to sidekiq:retry, see the retry timestamps (scores). Jobs scheduled for retry in 2 weeks? That's a lot of storage.

Scenario 3: Finding Specific Jobs

Need to find all jobs for a specific user?

CLI approach:

redis-cli LRANGE sidekiq:queue:default 0 -1 | grep "user_id\":123"

Works for small queues. For large queues, you're pulling everything.

Redimo approach:

1. Monitor sidekiq:queue:default
2. Jobs are displayed as a list
3. Each job payload is parsed and searchable
4. Filter/search within the results

Scenario 4: Bulk Retry Dead Jobs

Sidekiq Web lets you retry dead jobs one at a time, or all at once. But what if you want to retry only jobs of a specific class?

Ruby console approach:

Sidekiq::DeadSet.new.each do |job|
  job.retry if job.klass == "PaymentWorker"
end

Redis direct approach:

1. ZRANGE sidekiq:dead 0 -1 - get all dead jobs
2. Parse JSON, filter by class
3. Push matching jobs back to their queues
4. Remove from dead set

In Redimo, you can view dead jobs, filter visually, and bulk operate on selected items.

Worker Process Debugging

Active Processes

Monitor sidekiq:* and look at individual process hashes:

sidekiq:hostname:1234          # Hash

Fields:

• hostname - Server name
• pid - Process ID
• queues - Queues being worked
• concurrency - Thread count
• busy - Currently busy threads
• beat - Last heartbeat timestamp

Old heartbeats (stale processes) indicate workers that died without cleanup.

Clean Up Stale Processes

# Find stale process entries
SMEMBERS sidekiq:processes

# For each stale entry
SREM sidekiq:processes "hostname:1234:abc"
DEL sidekiq:hostname:1234:abc

Or use Sidekiq's built-in cleanup in the Web UI.

Statistics Deep Dive

Sidekiq stores stats in Redis:

sidekiq:stat:processed          # All-time processed
sidekiq:stat:failed             # All-time failed
sidekiq:stat:processed:2025-01-15  # Daily processed
sidekiq:stat:failed:2025-01-15     # Daily failed

Quick health check:

# Failure rate
MGET sidekiq:stat:processed sidekiq:stat:failed
# processed: 1000000, failed: 500
# 0.05% failure rate - healthy

Historical daily stats help identify patterns. Failures spike every Monday? Something's off with the weekly batch job.

Performance Tips

1. Use Separate Redis for Sidekiq

Don't share Redis between your app cache and Sidekiq. Queue operations are time-sensitive. Cache eviction shouldn't affect job processing.

2. Monitor Queue Latency

How long do jobs wait before processing?

# In a job
def perform(*args)
  enqueued_at = Time.at(self.class.jobs.last['enqueued_at'])
  latency = Time.now - enqueued_at
  StatsD.timing('sidekiq.latency', latency)
end

3. Set Reasonable Retry Limits

Default retry is 25 times over ~21 days. That's a long time for a broken job to consume resources.

class PaymentWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5  # Stop after 5 retries
end

4. Clean Up Dead Jobs Regularly

# Cron job or scheduled task
Sidekiq::DeadSet.new.clear
# Or keep last N
Sidekiq.options[:dead_max_jobs] = 1000

Redimo-Specific Features

When working with Sidekiq data in Redimo:

1. Pattern Monitor: sidekiq:* for overview, specific queues for debugging
2. Type awareness: Queues are Lists, retry/dead are Sorted Sets, processes use Sets + Hashes
3. JSON parsing: Job payloads render as expandable trees
4. Bulk operations: Select multiple jobs, delete, or export
5. Safe Mode: Prevent accidental deletions in production

Quick Reference

Sidekiq Concept	Redis Key	Redis Type
Queue	sidekiq:queue:<name>	List
Retry queue	sidekiq:retry	Sorted Set
Dead jobs	sidekiq:dead	Sorted Set
Scheduled	sidekiq:schedule	Sorted Set
Processes	sidekiq:processes	Set
Process info	sidekiq:<process-id>	Hash
Stats	sidekiq:stat:*	String (counter)

---

Sidekiq Web is great for most things. When you need to go deeper, understanding the Redis structure gets you answers faster than any abstraction layer.

Download Redimo and see your Sidekiq queues at the Redis level.