Celery Setup for macOS

Component: Celery (Task Queue for File Actions)
Deployment Method: Manual
Platform: macOS 12+
Author: Diskover Data, Inc.

Overview

Celery is the task queue that powers file actions in Diskover. When a user triggers a file action from the Diskover Web UI (such as copying, moving, or tagging files), that request is sent as a message through RabbitMQ and picked up by a Celery worker for execution. Without Celery running on your worker machine, file actions will not process.

This guide walks you through setting up Celery as a persistent background service on macOS using launchd — the native macOS service manager. On Linux, Celery runs under systemd, but macOS requires a different approach using LaunchDaemon .plist files.

When to use this guide:

• You have a macOS machine acting as a Diskover worker (scanner) and you need file actions to work on that machine

• You've already installed the Diskover worker stack (diskoverd) on macOS via the DMG installer or manually

Prerequisites

Verify Celery is installed:

Before proceeding, confirm that Celery is installed and accessible on your system:

which celery

You should see a path returned, for example:

/Library/Frameworks/Python.framework/Versions/3.x/bin/celery

If no path is returned, Celery is not installed. Install it via pip:

python3 -m pip install celery

Then run which celery again and note the path — you'll need it for the configuration file in the next section.

Verify the Celery application directory exists:

ls /opt/diskover/diskover_celery/

You should see the Diskover Celery application files. If this directory doesn't exist, the Diskover worker package may not be fully installed.

Installation

Step 1: Create the Celery Configuration File

The configuration file tells Celery where to find its application code, how to connect to RabbitMQ, and where to write logs. Create the file at /etc/celery.conf:

sudo vim /etc/celery.conf

Paste the following content, updating the placeholder values for your environment:

# Name of nodes to start
# here we have a single node
CELERYD_NODES="worker01"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute path to the 'celery' command:
# Use the output from 'which celery' for this value
CELERY_BIN="/Library/Frameworks/Python.framework/Versions/3.x/bin/celery"

# App instance to use
CELERY_APP="diskover_celery.worker"

# Path to the diskover installation directory
INSTALL_DIRECTORY="/opt/diskover"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
#CELERYD_OPTS="--time-limit=300 --concurrency=8"
CELERYD_OPTS="--without-heartbeat --without-mingle --without-gossip"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE='${INSTALL_DIRECTORY}/diskover_celery/run/%n.pid'
CELERYD_LOG_FILE='${INSTALL_DIRECTORY}/diskover_celery/log/%n%I.log'
CELERYD_LOG_LEVEL="INFO"

# Celery Beat options (optional)
CELERYBEAT_PID_FILE='${INSTALL_DIRECTORY}/diskover_celery/run/beat.pid'
CELERYBEAT_LOG_FILE='${INSTALL_DIRECTORY}/diskover_celery/log/beat.log'

Key values to update:

Save and close the file.

Step 2: Create the LaunchDaemon Plist

macOS uses launchd to manage background services (similar to systemd on Linux). You'll create a .plist file that tells macOS how to start and manage the Celery worker process.

Create the file at /Library/LaunchDaemons/com.diskover.celery.plist:

sudo vim /Library/LaunchDaemons/com.diskover.celery.plist

Paste the following content:

<plist version="1.0">
  <dict>
      <key>Label</key>
      <string>com.diskover.celery</string>

      <key>ProgramArguments</key>
      <array>
          <string>/bin/sh</string>
          <string>-c</string>
          <string>. /etc/celery.conf && cd ${INSTALL_DIRECTORY} && ${CELERY_BIN} -A ${CELERY_APP} worker --logfile=${CELERYD_LOG_FILE} --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS</string>
      </array>
      <key>UserName</key>
      <string>root</string>
      <key>GroupName</key>
      <string>wheel</string>
      <key>EnvironmentVariables</key>
      <dict>
          <key>PATH</key>
          <string>/usr/local/bin:/usr/bin:/bin</string>
      </dict>
      <key>RunAtLoad</key>
      <true/>
      <key>KeepAlive</key>
      <true/>
      <key>StandardOutPath</key>
      <string>/var/log/diskover-celery.log</string>
      <key>StandardErrorPath</key>
      <string>/var/log/diskover-celery.error.log</string>
  </dict>
  </plist>

What this plist does:

• RunAtLoad — Starts the Celery worker automatically when the Mac boots

• KeepAlive — Restarts the worker if it crashes or exits unexpectedly

• UserName / GroupName — Runs the service as root:wheel (required for access to /opt/diskover/)

• StandardOutPath / StandardErrorPath — Captures stdout and stderr to log files for troubleshooting

Step 3: Create Log and PID Directories

Celery needs directories to store its log files and process ID files. Create them if they don't already exist:

sudo mkdir -p /opt/diskover/diskover_celery/log
sudo mkdir -p /opt/diskover/diskover_celery/run

Step 4: Start the Celery Service

Load the LaunchDaemon to start the Celery worker:

sudo launchctl load /Library/LaunchDaemons/com.diskover.celery.plist

The service will start immediately and will also start automatically on future reboots.

Service Management

Note: Unlike systemctl restart on Linux, launchd does not have a single restart command. To restart Celery, unload and then load the plist:

sudo launchctl unload /Library/LaunchDaemons/com.diskover.celery.plist
sudo launchctl load /Library/LaunchDaemons/com.diskover.celery.plist

Verification

After starting the service, verify everything is working:

42967	0	com.diskover.celery

Configuration

Configuration File Locations

Tuning the Worker

You can adjust worker behavior by editing /etc/celery.conf:

• Concurrency — Uncomment and set --concurrency=N in CELERYD_OPTS to control how many tasks run in parallel (defaults to the number of CPU cores)

• Time limit — Uncomment --time-limit=300 to set a maximum execution time (in seconds) per task

• Log level — Change CELERYD_LOG_LEVEL to DEBUG for verbose output during troubleshooting, or WARNING for quieter production logging

• Multiple nodes — Change CELERYD_NODES to run multiple worker nodes (e.g., "w1 w2 w3")

After making changes, restart the service for them to take effect.

Troubleshooting

Log File Locations

Support

• https://support.diskoverdata.com

• https://support.diskoverdata.com/hc/en-us