Troubleshooting — Diskoverd Task-Worker

Overview

Diskoverd is the Diskover worker daemon. It runs on each indexer host and is responsible for:

• Receiving and executing crawl tasks — accepts jobs from the task queue and launches diskover crawl processes

• Worker coordination — registers the worker with the platform, reports health and capacity, and manages concurrent task execution

• Volume mounting — optionally mounts NFS/CIFS volumes before crawling and unmounts them after

• Scheduling — runs cron-based crawl schedules assigned to this worker

Diskoverd is a long-running Python process. It is typically started as a system service and kept running continuously alongside the Celery worker.

Service Management

Diskoverd is commonly run as a systemd service. If the service file has been installed:

RHEL / CentOS / Rocky Linux and Ubuntu

# Start
sudo systemctl start diskoverd

# Stop
sudo systemctl stop diskoverd

# Restart
sudo systemctl restart diskoverd

# Status
sudo systemctl status diskoverd

# Enable on boot
sudo systemctl enable diskoverd

Running Manually

Diskoverd can also be started directly for testing or troubleshooting:

cd /opt/diskover
python3 diskoverd.py

# With a custom worker name
python3 diskoverd.py -n my-worker-name

# Verbose logging to stdout
python3 diskoverd.py --verbose

# Skip NFS/CIFS mount capability checks (Linux workers without mount support)
python3 diskoverd.py --assumemountingenabled

Worker Identity

Each Diskoverd instance registers with the platform under a worker name. This name must be unique across all workers and is used to route tasks and identify the worker in the UI.

Worker name is resolved in this order:

1. -n <name> command-line option

2. DISKOVERD_WORKERNAME environment variable

3. Default: <hostname>_<5-char-unique-id> (automatically generated)

The worker name determines which Celery queues are consumed: index.<workername> and fileactions.<workername>. If the worker name changes between restarts, the old queues become orphaned in RabbitMQ.

Best practice: Set a stable, explicit worker name via -n or DISKOVERD_WORKERNAME so the queue names remain consistent across restarts.

Configuration

Worker configuration is managed through the Diskover Admin UI and stored in the shared SQLite database. No manual config file editing is required for most settings. A restart to any of the configs below require a restart of the ‘diskoverd’ service.

Navigate to Diskover Admin > Configuration > DiskoverD to adjust:

Logging Configuration

Log settings are under Diskover Admin > Configuration > DiskoverD > Logging:

Environment Variables

Any DiskoverD configuration field can be overridden via environment variable using the DISKOVERD_ prefix:

DISKOVERD_WORKERNAME=my-worker       # Override worker name
DISKOVERD_WORK_THREADS=8             # Override thread count

For volume encryption, set the encryption key via environment variable:

DISKOVER_ENCRYPT_SECRET_KEY=<key>    # Required for encrypted volume mounting

Log Locations

# Tail the main daemon log
sudo tail -f /var/log/diskover/diskoverd.log

# Tail the subprocess log (individual crawl output)
sudo tail -f /var/log/diskover/diskoverd_subproc.log

# Tail the service directly
journalctl -f -u diskoverd

Common Operations

Check Worker Status in the UI

Active workers and their health are visible in Diskover UI > Task Panel > Workers. A worker is considered offline if no heartbeat has been received for more than 10 minutes.

Check Worker is Registered

# Should show the worker's registered with the system
curl -XGET https://diskover-web-URL/diskover_admin/api/config/workers

Change Worker Concurrency

Increase or decrease Work Threads in Diskover Admin > Configuration > DiskoverD. Changes take effect after restarting diskoverd service on the appropriate worker.

A good starting point is one thread per physical CPU core. Increase for I/O-bound workloads (network storage), decrease if the indexer is CPU or memory constrained.

Run a Manual Crawl (Bypass Scheduler)

For ad-hoc crawls without going through the task queue, diskover can be invoked directly:

python3 /opt/diskover/diskover.py -i diskover-<indexname> /path/to/crawl

This bypasses diskoverd entirely and is useful for testing or one-off re-scans.

View Scheduled Tasks

Tasks assigned to this worker are visible in Diskover UI > Task Panel. The schedule, last run time, and status are shown for each task.

Troubleshooting

Worker Shows as Offline in the UI

Check the service is running:

sudo systemctl status diskoverd

Check the log for errors:

sudo journalctl -u diskoverd -n 100 --no-pager
sudo tail -50 /var/log/diskover/diskoverd.log

Common causes:

• Admin API unreachable — diskoverd fetches its config from diskover-admin on startup. If the API is down or the URL is misconfigured, the daemon will fail to start or will report as offline. Check the apiurl inside the /root/.config/diskoverd/config.yaml to ensure the Web host entry is correct.

• License failure — an expired or missing license causes diskoverd to exit at startup

• Wrong worker name — if the worker was previously registered under a different name, it may appear as a second (offline) worker in the UI. Use -n to enforce a consistent name

Tasks Not Running on This Worker

Tasks are dispatched to a specific worker based on the worker name name inside the task definition. If tasks aren't running:

1. Confirm the task is assigned to the correct worker name:

   1. Edit the task

   2. Check Assigned Worker has the correct worker selected

2. Check the worker service is running — diskoverd must be running for crawl tasks to execute:

   sudo systemctl status diskoverd

3. Check work thread capacity — if all Work Threads slots are occupied by running tasks, new tasks will queue. Check active tasks in Diskover UI > Task Panel > Workers.

Crawl Subprocess Fails Immediately

The crawl process is launched by diskoverd but exits before completing.

sudo tail -100 /var/log/diskover/diskoverd_subproc.log

Common causes:

• Elasticsearch unreachable from this worker — test: curl -s http://<es-host>:9200/_cluster/health

• Mount point not accessible — verify the path exists and is readable

• Python path issue — verify python3 /opt/diskover/diskover.py --version runs successfully

NFS/CIFS Mount Failures

diskoverd mounts volumes automatically before crawling if volume mounting is configured.

# Check NFS/CIFS availability on the worker
showmount -e <nfs-server>         # NFS
smbclient -L //<cifs-server>/ -N  # CIFS

If the worker machine does not have NFS/CIFS client packages installed, either:

• Install the required packages (nfs-utils on RHEL/Rocky, nfs-common on Ubuntu; cifs-utils on both)

• Or start diskoverd with --assumemountingenabled to skip the capability check (only use if mounting is handled externally)

High Memory or CPU Usage

diskoverd launches a subprocess per concurrent task. If Work Threads is set high and tasks are I/O-intensive:

• Reduce Work Threads in Diskover Admin > Configuration > DiskoverD

• Monitor subprocess memory: ps aux | grep diskover

• Check if any subprocesses are hung: a stuck subprocess will hold a thread slot indefinitely

To kill a hung crawl subprocess without stopping diskoverd:

# Find the subprocess PID
ps aux | grep diskover.py

# Kill the specific subprocess
sudo kill -TERM <pid>

diskoverd will detect the subprocess exit and free the thread slot.