Troubleshooting — RabbitMQ

Overview

RabbitMQ is the message broker for the Diskover platform. It acts as the communication layer between diskover-admin (which submits jobs) and the Celery workers running on indexer machines (which execute them). Every crawl task and file action passes through RabbitMQ.

Diskover uses the AMQP protocol (pyamqp) to connect to RabbitMQ. The broker URL format is:

pyamqp://<user>:<password>@<host>//

Service Management

RHEL / CentOS / Rocky Linux

# Start
sudo systemctl start rabbitmq-server

# Stop
sudo systemctl stop rabbitmq-server

# Restart
sudo systemctl restart rabbitmq-server

# Status
sudo systemctl status rabbitmq-server

# Enable on boot
sudo systemctl enable rabbitmq-server

Ubuntu / Debian

# Start
sudo systemctl start rabbitmq-server

# Stop
sudo systemctl stop rabbitmq-server

# Restart
sudo systemctl restart rabbitmq-server

# Status
sudo systemctl status rabbitmq-server

# Enable on boot
sudo systemctl enable rabbitmq-server

Configuration

Diskover Broker Settings

RabbitMQ connection details are configured in Diskover Admin > Configuration > Task Queue:

Important: Change the default password (darkdata) in any production environment. Update it in both RabbitMQ and in Diskover Admin > Configuration > Task Queue.

RabbitMQ User Setup

If setting up a fresh RabbitMQ instance for Diskover:

# Add the Diskover user
sudo rabbitmqctl add_user diskover <password>

# Grant full permissions on the default vhost
sudo rabbitmqctl set_permissions -p / diskover ".*" ".*" ".*"

# Optional: grant management UI access
sudo rabbitmqctl set_user_tags diskover management

Enable the Management Plugin

The management plugin provides a web UI on port 15672 and the rabbitmqctl HTTP API:

sudo rabbitmq-plugins enable rabbitmq_management

# Restart to activate
sudo systemctl restart rabbitmq-server

Access the management UI at http://<host>:15672. Log in with the diskover user or the default guest account (localhost only).

Log Locations

sudo tail -f /var/log/rabbitmq/rabbit@$(hostname).log

Common Operations

Check RabbitMQ Status

sudo rabbitmqctl status

List Queues

Diskover creates two queues per worker node, named after the worker's hostname:

• index.<hostname> — crawl/indexing tasks (supports priority 0–10)

• fileactions.<hostname> — file operation tasks

sudo rabbitmqctl list_queues name messages consumers

Or via the management API:

curl -s -u diskover:<password> http://localhost:15672/api/queues | python3 -m json.tool

Check Queue Depth (Backlog)

sudo rabbitmqctl list_queues name messages

A queue with a growing messages count and no consumers means the associated Celery worker is not running or not consuming. See the Celery doc.

Purge a Queue

Use with caution — this permanently discards all pending messages in a queue:

sudo rabbitmqctl purge_queue index.<hostname>

Or from the management UI: Queues tab → select queue → Purge Messages.

List Connections and Channels

sudo rabbitmqctl list_connections user peer_host peer_port state
sudo rabbitmqctl list_channels connection number messages_unacknowledged

Check Exchanges

Diskover uses two direct exchanges: index and fileactions. To verify they exist:

sudo rabbitmqctl list_exchanges name type

List Users and Permissions

sudo rabbitmqctl list_users
sudo rabbitmqctl list_permissions -p /

Change a User Password

sudo rabbitmqctl change_password diskover <new_password>

Then update the password in Diskover Admin > Configuration > Task Queue to match.

Troubleshooting

Celery Workers Cannot Connect to RabbitMQ

Check RabbitMQ is running:

sudo systemctl status rabbitmq-server

Test connectivity from the worker machine:

# Basic port check
nc -zv <rabbitmq-host> 5672

# Or with curl (management API)
curl -s -u diskover:<password> http://<rabbitmq-host>:15672/api/overview

Check firewall — port 5672 (AMQP) must be open:

# RHEL / Rocky
sudo firewall-cmd --list-all
sudo firewall-cmd --add-port=5672/tcp --permanent
sudo firewall-cmd --reload

# Ubuntu
sudo ufw status
sudo ufw allow 5672/tcp

Verify credentials in Diskover Admin > Configuration > Task Queue match the RabbitMQ user.

Messages Accumulating / No Consumers

Queues show messages but consumers count is 0. The Celery worker for that hostname is down.

# Check worker service on the indexer machine
sudo systemctl status celery

# Check which queues are registered and which workers are consuming
sudo rabbitmqctl list_consumers

RabbitMQ Won't Start

sudo journalctl -u rabbitmq-server -n 100 --no-pager

Common causes:

• Erlang not installed or version mismatch — RabbitMQ requires a compatible Erlang version. Check the RabbitMQ Erlang compatibility matrix.

• Port conflict on 5672 or 15672 — check with sudo ss -tlnp | grep '5672\|15672'

• Hostname resolution — RabbitMQ is sensitive to hostname changes. Ensure the system hostname is stable and resolvable.

Authentication Failure (ACCESS_REFUSED)

Celery worker logs show ACCESS_REFUSED or authentication failure.

1. Verify the user exists: sudo rabbitmqctl list_users

2. Verify permissions: sudo rabbitmqctl list_permissions -p /

3. Confirm the password in Diskover Admin > Configuration > Task Queue matches RabbitMQ

4. Confirm the vhost is / (double slash in broker URL pyamqp://user:pass@host// is correct — the trailing // specifies the default vhost)

Official Documentation

• RabbitMQ Documentation

• RabbitMQ: Access Control