Breadcrumb Index Plugin User Guide

License: PRO+ (Professional Edition or higher)
Plugin Type: Index Plugin
Author: Diskover Data, Inc.

Overview

The Breadcrumb plugin extracts metadata from breadcrumb files during Diskover indexing and adds this information to directory documents in your Elasticsearch index. Breadcrumb files are small text files containing key-value metadata about a directory—typically used to track project ownership, creation dates, storage quotas, and other organizational metadata.

When enabled, this plugin automatically reads breadcrumb files as directories are crawled, making their metadata searchable alongside all your other file system data. This allows you to answer questions like "Which directories belong to the Engineering department?" or "Show me all project directories created before 2023" directly from the Diskover search interface.

Key Capabilities

• Automatic Metadata Extraction: Reads key-value pairs from breadcrumb files and adds them to directory documents during indexing

• Configurable Structure: Customize the breadcrumb directory name, file name, and target field name to match your existing conventions

• Auto-Tagging: Automatically tags directories containing breadcrumb data for easy filtering

• Date Normalization: Converts birthdate fields from dd/mm/yyyy to yyyy-mm-dd format for proper date searching

• Field Filtering: Optionally extract only specific fields from breadcrumb files

• Performance Caching: SQLite caching improves re-scan performance for large environments

Use Cases

For Storage Administrators

For Data Managers and Project Leads

Understanding Breadcrumb Files

Before configuring this plugin, it helps to understand what breadcrumb files are and how they work. This section explains the concept for users who may be new to this approach.

What Are Breadcrumb Files?

Breadcrumb files are simple text files that contain metadata about a directory in a key-value format. Organizations typically create these files during directory provisioning workflows to record important information such as:

• When the directory was created (birthdate)

• Who owns or requested the directory

• Which project or department the directory belongs to

• Quota or storage allocation information

• Ticket or request references for audit trails

The term "breadcrumb" comes from the idea of leaving a trail of information—these files provide context about why a directory exists and who is responsible for it.

Typical Directory Structure

Breadcrumb files are usually stored in a hidden subdirectory within the target directory. Here's an example structure:

/mnt/projects/
└── alpha_project/
    ├── .birthday/            ← Breadcrumb directory (hidden)
    │   └── volinfo           ← Breadcrumb file with metadata
    ├── data/
    ├── scripts/
    └── results/

In this example, the .birthday directory contains a file called volinfo with metadata about the alpha_project directory.

Breadcrumb File Format

Breadcrumb files use a simple key: value text format with one entry per line. Here's an example breadcrumb file:

birthdate: 15/03/2024
owner: sarah.chen
project: AlphaResearch
department: DataScience
quota_gb: 500
requestor: mike.johnson
cost_center: CC-4521
ticket: STOR-78432

Each line contains a field name, followed by a colon, followed by the value. The plugin reads these pairs and makes them searchable in Diskover.

Special Field Handling

Why Use Breadcrumb Files?

Requirements

License

License: PRO+ (Professional Edition or higher)

Python Dependencies

This plugin has no external Python package dependencies beyond Diskover's core requirements.

System Requirements

• Python 3.9 or higher

• Diskover indexer with plugin support enabled

• Read access to breadcrumb files during indexing

• Write access to cache directory (if caching is enabled)

Breadcrumb File Requirements

For the plugin to successfully extract metadata, breadcrumb files must meet these criteria:

• Text file with UTF-8 encoding

• One key-value pair per line in key: value format

• Non-empty file (empty files are skipped with a warning)

Installation

Step 1: Install Dependencies

This plugin has no additional dependencies to install.

Step 2: Configure the Plugin

1. Navigate to Diskover Admin > Plugins > Index Plugins > Breadcrumb

2. Enable the plugin

3. Configure the parameters to match your breadcrumb file structure (see Configuration section below)

4. Save your configuration

Step 3: Enable in Index Task Configuration

1. Navigate to Diskover > Configurations > select your configuration (e.g., Default)

2. Scroll to the bottom to find Index Plugins Enablement

3. Enable the Breadcrumb plugin

4. Save the configuration

The plugin will now run automatically during any scan that uses this configuration.

Step 4: Create Breadcrumb Files (If Not Already Present)

If your environment doesn't already have breadcrumb files, you can create them manually or through automated provisioning scripts.

Linux Example:

# Create breadcrumb directory
mkdir -p /mnt/projects/my_project/.birthday

# Create breadcrumb file with metadata
cat > /mnt/projects/my_project/.birthday/volinfo << 'EOF'
birthdate: 15/03/2024
owner: sarah.chen
project: AlphaResearch
department: DataScience
quota_gb: 500
requestor: mike.johnson
cost_center: CC-4521
ticket: STOR-78432
EOF

Windows Example:

# Create breadcrumb directory
New-Item -ItemType Directory -Path "D:\Projects\my_project\.birthday" -Force

# Create breadcrumb file with metadata
@"
birthdate: 15/03/2024
owner: sarah.chen
project: AlphaResearch
department: DataScience
quota_gb: 500
requestor: mike.johnson
cost_center: CC-4521
ticket: STOR-78432
"@ | Out-File -FilePath "D:\Projects\my_project\.birthday\volinfo" -Encoding UTF8

Configuration

Configuration Parameters

Default Cache Directories:

• Linux: /opt/diskover/__breadcrumb_plugin_cache__/

• Windows: C:\Program Files\Diskover\__breadcrumb_plugin_cache__\

Understanding Key Parameters

breadcrumb_dir_name

This setting specifies the subdirectory that contains the breadcrumb file. Common configurations:

target_field_name

The Elasticsearch field name determines how you'll search for breadcrumb data in Diskover:

extract_fields

Control which fields are extracted from breadcrumb files:

Example Configurations

Default Configuration

Standard breadcrumb extraction with all fields:

{
  "breadcrumb_dir_name": ".birthday",
  "breadcrumb_file_name": "volinfo",
  "target_field_name": "volinfo",
  "breadcrumb_dir_parent_tag": "quotadir",
  "enable_cache": true,
  "cache_dir": "/opt/diskover/__breadcrumb_plugin_cache__/",
  "cache_expire_time": 0,
  "extract_fields": []
}

Storage Quota Tracking Configuration

Focused on quota and ownership metadata:

{
  "breadcrumb_dir_name": ".birthday",
  "breadcrumb_file_name": "volinfo",
  "target_field_name": "storage_info",
  "breadcrumb_dir_parent_tag": "quota-tracked",
  "enable_cache": true,
  "cache_dir": "/opt/diskover/__breadcrumb_plugin_cache__/",
  "cache_expire_time": 0,
  "extract_fields": ["owner", "department", "quota_gb", "birthdate"]
}

Windows-Compatible Configuration

Using visible directory names for Windows environments:

{
  "breadcrumb_dir_name": "_metadata",
  "breadcrumb_file_name": "dirinfo.txt",
  "target_field_name": "dirinfo",
  "breadcrumb_dir_parent_tag": "has-metadata",
  "enable_cache": true,
  "cache_dir": "C:\\Program Files\\Diskover\\__breadcrumb_plugin_cache__\\",
  "cache_expire_time": 0,
  "extract_fields": []
}

Direct File Configuration

When breadcrumb files are directly in the parent directory (no subdirectory):

{
  "breadcrumb_dir_name": "",
  "breadcrumb_file_name": ".volinfo",
  "target_field_name": "volinfo",
  "breadcrumb_dir_parent_tag": "quotadir",
  "enable_cache": true,
  "cache_dir": "/opt/diskover/__breadcrumb_plugin_cache__/",
  "cache_expire_time": 0,
  "extract_fields": []
}

Indexed Fields / Elasticsearch Mappings

The Breadcrumb plugin adds a configurable object field (default: volinfo) to directory documents containing the extracted metadata.

Field Mappings

Example Indexed Document

Given the example breadcrumb file from earlier:

birthdate: 15/03/2024
owner: sarah.chen
project: AlphaResearch
department: DataScience
quota_gb: 500
requestor: mike.johnson
cost_center: CC-4521
ticket: STOR-78432

The resulting directory document in Elasticsearch would include:

{
  "name": "alpha_project",
  "type": "directory",
  "path": "/mnt/projects/alpha_project",
  "tags": ["quotadir"],
  "volinfo": {
    "birthdate": "2024-03-15",
    "owner": "sarah.chen",
    "project": "AlphaResearch",
    "department": "DataScience",
    "quota_gb": "500",
    "requestor": "mike.johnson",
    "cost_center": "CC-4521",
    "ticket": "STOR-78432",
    "breadcrumb_file_path": "/mnt/projects/alpha_project/.birthday/volinfo"
  }
}

Notice that the birthdate value has been automatically converted from 15/03/2024 to 2024-03-15 for proper date handling.

Searching in Diskover

Once indexing is complete, you can search for directories based on their breadcrumb metadata. The examples below assume the default target_field_name of volinfo and use the sample breadcrumb file shown earlier.

Basic Searches

Find all directories with breadcrumb metadata:

volinfo:*

Find directories tagged as quota directories:

tags:quotadir

Searching by Owner

Searching by Project

Searching by Department

Searching by Cost Center

Searching by Quota

Searching by Ticket Reference

Searching by Birthdate

Combined Searches

These queries combine multiple criteria for more targeted results:

Storage Analysis Searches

Troubleshooting

Common Issues

Verifying Breadcrumb File Setup

Check if the breadcrumb directory exists:

Linux:

ls -la /path/to/directory/.birthday/

Windows:

Get-ChildItem -Path "D:\path\to\directory\.birthday" -Force

Check if the breadcrumb file is readable and has content:

Linux:

cat /path/to/directory/.birthday/volinfo
wc -l /path/to/directory/.birthday/volinfo

Windows:

Get-Content "D:\path\to\directory\.birthday\volinfo"

Validate file encoding:

Linux:

file /path/to/directory/.birthday/volinfo

The output should indicate UTF-8 or ASCII text.

Cache Management

If you experience cache-related issues, you can clear the cache to force re-extraction of breadcrumb data.

Linux:

rm -rf /opt/diskover/__breadcrumb_plugin_cache__/

Windows:

Remove-Item -Recurse -Force "C:\Program Files\Diskover\__breadcrumb_plugin_cache__\"

Fix cache directory permissions (Linux):

mkdir -p /opt/diskover/__breadcrumb_plugin_cache__
chown diskover:diskover /opt/diskover/__breadcrumb_plugin_cache__
chmod 755 /opt/diskover/__breadcrumb_plugin_cache__

Debug Logging

To troubleshoot issues, monitor the Diskover log for breadcrumb-related messages:

Linux:

tail -f /var/log/diskover/diskover.log | grep -i breadcrumb

Windows:
Check the Diskover service logs or configured log location for entries containing "breadcrumb".

Test Breadcrumb File Parsing

You can manually test if a breadcrumb file will parse correctly:

# Save as test_breadcrumb.py and run with: python3 test_breadcrumb.py
path = "/path/to/directory/.birthday/volinfo"
with open(path, 'r') as f:
    for line in f:
        line = line.strip()
        if line:
            idx = line.find(':')
            if idx > 0:
                name = line[:idx].strip()
                val = line[idx+1:].strip()
                print(f"Field: {name} = {val}")
            else:
                print(f"Invalid line (no colon found): {line}")

Support

• Diskover Documentation

• Diskover Support Portal

Last Updated: January 2026
Diskover Data, Inc.