SpectraLogic / RioBroker

License: PRO+ (Professional Edition or higher)
Module Type: Alternate Scanner
Author: Diskover Data, Inc.

Overview

The Diskover SpectraLogic / RioBroker Scanner indexes object storage managed by Spectra Logic's RioBroker middleware directly into Diskover. If your organization relies on Spectra Logic BlackPearl systems for tape-based archival or tiered object storage, this scanner gives you full visibility into that data — searchable, analyzable, and manageable through the Diskover Web UI, just like any other indexed storage.

The scanner connects to the RioBroker REST API, enumerates your buckets, and converts the object hierarchy into standard Diskover documents in Elasticsearch. Once indexed, your archived data appears alongside all your other storage in Diskover's file tree, search, and analytics views.

Why Use This Scanner?

• Archive and Tape Storage Indexing — See everything stored across your BlackPearl tape libraries without needing to access tape systems directly. Search, tag, and analyze archived data as easily as active storage.

• Storage Capacity Analysis — Understand how storage is distributed across RioBroker-managed buckets. Identify large files, analyze growth trends, and plan capacity with real data.

• Multi-Tier Storage Management — Combine this scanner with other Diskover scanners to create a unified view of data across active, nearline, and deep archive tiers. One search interface for all your storage.

• Data Migration Planning — Build a complete inventory of objects before migrating from BlackPearl to another platform. Understand file types, sizes, and distribution patterns to plan effectively.

• Compliance and Data Governance — Maintain searchable indexes of compliance-related files in archival storage. Respond to audits and governance requests efficiently using Diskover's tagging and search capabilities.

• Sample metadata from a Spectra / Rio Broker scan:

With each file / folder in the Spectra scan we get these additional metadata fields if they exists on the Rio Broker API for that object!

Understanding Spectra Logic RioBroker

Spectra Logic's RioBroker is a middleware layer that provides RESTful API access to BlackPearl object storage systems. It sits between your applications and the underlying BlackPearl hardware (tape libraries, disk storage), abstracting the complexity of managing data across different storage tiers.

Here are a few key concepts that will help you configure the scanner:

Key Concepts

How the Scanner Works

When you run a scan, the scanner:

1. Integrates as an Alternate Scanner with the core Diskover scanner

2. Authenticates with the RioBroker API and obtains a JWT token

3. Enumerates objects and directories within your specified bucket

4. Converts each object into a Diskover-compatible document with standard file metadata and Spectra-specific fields

5. Hands documents to Diskover's core indexing pipeline for Elasticsearch upload

6. Automatically refreshes the authentication token during long-running scans

The result is a fully searchable Diskover index that reflects the contents of your BlackPearl storage, complete with file sizes, directory structures, custom metadata fields, and automatic tagging.

Requirements

System Requirements

External Service Requirements

Python Dependencies

Installation

Step 1: Install the Scanner Package

Linux (RPM):

dnf install diskover-scanner-spectra

Windows:
The scanner files are included with the Diskover Windows installation. No separate installation step is required.

Install locations:

Step 2: Install Python Dependencies

Navigate to the scanner directory and install the required packages:

Linux:

cd /opt/diskover/scanners/scandir_spectra
python3 -m pip install -r etc/requirements.txt

Windows:

cd "C:\Program Files\Diskover\scanners\scandir_spectra"
python -m pip install -r etc\requirements.txt

Note: On Linux, the uvloop package is installed automatically for improved async performance. On Windows, the scanner uses a native high-resolution event loop instead — no additional steps are needed.

Step 3: Verify RioBroker Connectivity

Before running a scan, confirm that the scanner can reach the RioBroker API. Test the connection with a direct API call:

curl -k -X 'POST' \
  'https://YOUR_RIOBROKER_HOST:5050/api/tokens' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "username": "spectra",
    "password": "spectra"
  }'

A successful response returns a JWT token, confirming API access. If this fails, check your network connectivity, firewall rules (port 5050 must be open for HTTPS), and credentials.

Step 4: Test the Scanner Connection

The scanner includes a test utility that authenticates with RioBroker and lists all available brokers and buckets:

Linux:

cd /opt/diskover/scanners/scandir_spectra
python3 test.py

Windows:

cd "C:\Program Files\Diskover\scanners\scandir_spectra"
python test.py

Important: The test.py script uses hardcoded credentials. Before running it, edit the rio_broker_user, rio_broker_password, rio_broker_api_domain, and rio_broker_api_port variables at the top of test.py to match your environment.

Expected output shows each broker and its associated bucket:

2024-01-15 10:30:45 - spectra_scanner - INFO - Broker: {'name': 'broker1', ...} Bucket: archive-bucket
2024-01-15 10:30:46 - spectra_scanner - INFO - Broker: {'name': 'broker2', ...} Bucket: backup-bucket

Configuration

The Spectra scanner is configured through the Diskover Admin UI. Navigate to Diskover > Alternate Scanners > Spectra to access the configuration panel.

Configuration via Diskover Admin

1. Navigate to Diskover Admin > Diskover > Alternate Scanners > Spectra

2. Enter your RioBroker connection details (hostname/IP, port, credentials)

3. Adjust performance tuning parameters as needed

4. Save the configuration

Sample Configuration in Diskover Admin:

Here is the beginning of our sample configuration There are many other configuraitons for the Spectra Scanner - covered in detail below!

Configuration Parameters

Warning: Increasing concurrent_request_limit and request_limit_per_minute beyond defaults may destabilize the RioBroker server. Test changes incrementally in non-production environments and monitor RioBroker server health.

Configuration Examples

Example 1: Standard Configuration

A typical production setup connecting to a dedicated RioBroker server:

Example 2: Conservative / Shared-Server Configuration

When the RioBroker server is shared with other applications or under heavy load, use lower concurrency settings:

Usage

The Spectra scanner integrates with Diskover using the standard --altscanner flag. It replaces Diskover's default filesystem walker with one that reads from RioBroker-managed buckets.

Command Syntax

Linux:

cd /opt/diskover
python3 diskover.py --altscanner scandir_spectra <bucket_name>/<path>

Windows:

cd "C:\Program Files\Diskover"
python diskover.py --altscanner scandir_spectra <bucket_name>/<path>

Path Format Reference

The scan path uses the format <bucket_name>/<optional_subpath>. The bucket name is the first component of the path, and the scanner automatically maps it into Diskover's file tree.

Usage Examples

Basic Single Bucket Scan

Index an entire bucket with default settings:

Linux:

cd /opt/diskover
python3 diskover.py --altscanner scandir_spectra archive-bucket

Windows:

cd "C:\Program Files\Diskover"
python diskover.py --altscanner scandir_spectra archive-bucket

Scan a Specific Subdirectory

Index only a specific path within a bucket:

python3 diskover.py --altscanner scandir_spectra archive-bucket/2024/reports

Custom Index Name

Specify a custom Elasticsearch index name:

python3 diskover.py --altscanner scandir_spectra -i diskover-spectra-archive archive-bucket

Verbose / Debug Logging

Enable verbose logging for troubleshooting:

python3 diskover.py --altscanner scandir_spectra --loglevel DEBUG archive-bucket

Example output from a Spectra CLI Scan:

python3 /opt/diskover/diskover.py --altscanner spectra -i diskover-spectra-graphics-archive graphics


diskover - INFO - Using alternate scanner spectra
scanners.scandir_spectra.scandir_spectra - INFO - Initializing Spectra scanner
scanners.scandir_spectra.scandir_spectra - INFO - Checking Spectra path: bucket=graphics, path=/
scanners.scandir_spectra.scandir_spectra - INFO - Scanning Spectra directory: graphics
scanners.scandir_spectra.spectra_scanner.spectra_rio_broker.client - INFO - Graph request limit per minute: 6000, concurrency limit: 10
scanners.scandir_spectra.spectra_scanner.spectra - INFO - Initializing Spectra client with 8 workers
diskover - INFO - configuration Default
diskover - INFO - Creating index diskover-spectra-graphics-archive...
diskover_elasticsearch - INFO - Tuning index settings for crawl
diskover - INFO - No plugins loaded
diskover - INFO - maxwalkthreads set to 8
diskover - INFO - maxthreads set to 8
diskover - INFO - maxthreaddepth set to 999
diskover - INFO - indexthreads set to 16
diskover - INFO - Enqueuing dir tree graphics
scanners.scandir_spectra.scandir_spectra - INFO - Scanning Spectra directory: graphics
scanners.scandir_spectra.scandir_spectra - INFO - Scanning Spectra directory: graphics/GFX_Main
diskover - INFO - 1 paths still scanning ['graphics/GFX_Main'], memory usage 98.36 MB)

Integration with Index Tasks

The Spectra scanner can be configured as part of a Diskover Index Task for scheduled or automated scanning.

When configuring an Index Task in the Diskover Admin UI, set the Alternate Scanner field to scandir_spectra and specify the bucket path as the scan target.

Example of a Diskover Index Task Scan:

Performance Tips

• Start conservative — Begin with the default settings and increase concurrency only after confirming the RioBroker server handles the load well.

• Tune walk_workers — This controls how fast the scanner traverses directories. Increase for large, deep directory trees; decrease if the RioBroker server is under load.

• Adjust concurrent_request_limit carefully — Higher concurrency speeds up scanning but can destabilize the RioBroker server. Monitor server health when increasing this value.

• Use log_below_depth — Set this to a low number in production to reduce log noise. Set to 999 during troubleshooting to see all directory traversal activity.

Metadata Fields / Elasticsearch Mappings

The Spectra scanner adds several custom metadata fields to each indexed document beyond standard Diskover file metadata. These fields are automatically mapped in Elasticsearch when the scanner runs.

Field Mappings

Elasticsearch Mapping Definition

{
  "mappings": {
    "properties": {
      "bucket": {
        "type": "keyword"
      },
      "spectra_owner": {
        "type": "keyword"
      },
      "spectra_group": {
        "type": "keyword"
      },
      "spectra_mode": {
        "type": "integer"
      },
      "creation_date": {
        "type": "date"
      }
    }
  }
}

Automatic Tagging

The scanner automatically tags every indexed file with:

• bucket:<bucket_name> — Identifies which RioBroker bucket the file belongs to

• storage:spectra — Identifies the file as originating from Spectra storage

These tags appear in the Diskover Web UI and can be used in searches and filters.

Searching in Diskover

Once a scan completes, the Spectra-specific metadata fields and tags are available for searching in the Diskover Web UI. Here are some common search examples.

Search Query Examples

Combining with Standard Diskover Search

Spectra metadata fields work alongside all standard Diskover search capabilities. You can combine them with path filters, file type filters, size ranges, and tags to build powerful queries across your entire indexed storage landscape.

Here we can see some sample metadata in Diskover from an object scanned by the Spectra scanner:

Troubleshooting

Common Issues

Debug Logging

Enable debug logging using the standard Diskover log level flag:

python3 /opt/diskover/diskover.py --altscanner scandir_spectra --loglevel DEBUG archive-bucket

Log File Locations:

• Linux: /var/log/diskover/diskover.log

• Windows: Check Diskover service logs or configured log location

Verifying Connectivity

Test RioBroker API access:

curl -k -X POST 'https://RIOBROKER_HOST:5050/api/tokens' \
  -H 'Content-Type: application/json' \
  -d '{"username": "YOUR_USER", "password": "YOUR_PASS"}'

Test scanner connectivity with the built-in test utility:

cd /opt/diskover/scanners/scandir_spectra
python3 test.py

Support

• Diskover Documentation

• Diskover Support Portal

• Spectra Logic Documentation

Last Updated: April 2026
Diskover Data, Inc.