First Index Time

License: PRO+ (Professional Edition or higher)
Plugin Type: Index Plugin
Author: Diskover Data, Inc.

Overview

The First Index Time plugin records the timestamp when files first appear in Diskover's index—commonly called the "arrival date" or "first index time." This metadata is invaluable for tracking when data enters your storage environment, independent of filesystem timestamps that may change during migrations, backups, or file modifications.

Unlike filesystem timestamps (ctime, mtime, atime), the first index time represents when Diskover first discovered and indexed a file. This provides an objective arrival date that remains consistent regardless of how the file was created or subsequently modified.

Key Capabilities

• Persistent Tracking: Records the first-seen timestamp in a SQLite cache that persists across re-indexes

• Inode Validation: Uses inode numbers to detect file replacements (same path, different file)

• Directory Filtering: Include or exclude specific directories using exact paths or regex patterns

• UTC Timestamps: Stores all timestamps in ISO 8601 format with UTC timezone for consistency

Use Cases

For Storage Administrators:

For Data Governance and Compliance:

For Business Workflows:

Indexed Field

This plugin adds a single field to indexed documents:

Understanding First Index Time

What is First Index Time?

First index time is the timestamp recorded when Diskover first encounters and indexes a file. This is fundamentally different from filesystem timestamps:

Why Filesystem Timestamps Are Insufficient

Filesystem timestamps are often unreliable for tracking when data actually arrived in your storage:

• File Migrations: Moving files between storage systems often resets or alters timestamps

• Backup Restorations: Restored files may carry original timestamps from years ago

• Archive Extraction: Extracted files inherit timestamps from when they were originally created

• Copy Operations: Many copy tools preserve source timestamps rather than recording transfer time

• Application Behavior: Some applications intentionally set timestamps to specific values

The first index time provides a reliable "arrival date" that reflects when data actually entered your managed storage environment.

How Inode Validation Works

The plugin uses inode numbers to ensure data integrity and detect file replacements:

1. Initial Index: Plugin records the file path, inode number, and current timestamp

2. Re-Index: Plugin looks up the cached entry by file path

3. Validation: If the stored inode matches the current inode, the cached timestamp is returned

4. Replacement Detection: If the inode differs (file was replaced), a new timestamp is recorded

This approach correctly handles scenarios where a file is deleted and replaced with a new file at the same path—the new file receives a new first index time, reflecting that it's genuinely a different file.

Requirements

System Requirements

• Python 3.9 or higher

• Diskover PRO+ license or higher

• Write access to cache directory for SQLite database

• Sufficient disk space for cache (minimal—approximately 100 bytes per unique file)

Python Dependencies

This plugin has no external Python dependencies beyond the Diskover core libraries. All required functionality is provided by Python's standard library.

Installation

Step 1: Configure the Plugin

1. Navigate to Diskover Admin → Plugins → Index Plugins

2. Locate First Index Time in the plugin list

3. Enable the plugin and configure parameters as needed

4. Save the configuration

Step 2: Enable in Index Task Configuration

1. Navigate to Diskover → Configurations → select your configuration (e.g., Default)

2. Scroll to the bottom and locate Index Plugins Enablement

3. Enable the First Index Time plugin

4. Save the configuration

The plugin will now run automatically during scans using this configuration.

Step 3: Verify Cache Directory

Ensure the cache directory exists and is writable by the Diskover service account:

Linux:

# Check default location
ls -la /opt/diskover/__diskover_firstindextime_plugin_cache__/

# Create if needed
mkdir -p /opt/diskover/__diskover_firstindextime_plugin_cache__
chown diskover:diskover /opt/diskover/__diskover_firstindextime_plugin_cache__
chmod 755 /opt/diskover/__diskover_firstindextime_plugin_cache__

Windows:

# Check default location
dir "C:\Program Files\Diskover\__diskover_firstindextime_plugin_cache__"

# Create if needed (run as Administrator)
mkdir "C:\Program Files\Diskover\__diskover_firstindextime_plugin_cache__"

Configuration

Configuration Parameters

Configuration Examples

Standard Configuration (Recommended)

Track all files with permanent cache retention:

{
  "verbose": false,
  "cachedir": "/opt/diskover/__diskover_firstindextime_plugin_cache__/",
  "cache_expiretime": 0,
  "include_dirs": [],
  "exclude_dirs": []
}

Project-Specific Tracking

Track only files in specific project directories:

{
  "verbose": false,
  "cachedir": "/opt/diskover/__diskover_firstindextime_plugin_cache__/",
  "cache_expiretime": 0,
  "include_dirs": [
    "/mnt/data/projects",
    "/mnt/data/deliverables"
  ],
  "exclude_dirs": []
}

Exclude Temporary Directories

Exclude scratch and temporary directories from tracking:

{
  "verbose": false,
  "cachedir": "/opt/diskover/__diskover_firstindextime_plugin_cache__/",
  "cache_expiretime": 0,
  "include_dirs": [],
  "exclude_dirs": [
    "/mnt/data/temp",
    "/mnt/data/scratch",
    ".*\\.tmp$",
    ".*cache.*"
  ]
}

Understanding Directory Filters

The include_dirs and exclude_dirs parameters support multiple matching modes:

Note: Directory exclusion takes precedence over inclusion. If a path matches both lists, it is excluded.

Indexed Fields / Elasticsearch Mappings

Field Mapping

Field Characteristics

• Date type: Enables date range queries and date histogram aggregations

• ISO 8601 format: Example: 2025-03-15T14:30:00

• UTC timezone: All timestamps are stored in UTC for consistency

• Immutable: Once recorded, the value never changes for that file (unless the inode changes)

Example Document

A file indexed with the First Index Time plugin:

{
  "name": "quarterly_report.pdf",
  "extension": "pdf",
  "size": 2048576,
  "mtime": "2025-01-10T09:15:22",
  "firstindextime": "2025-03-15T14:30:00"
}

Note how firstindextime differs from mtime. The file was last modified in January but first appeared in Diskover's index in March—this could indicate the file was copied or migrated from another location.

Searching in Diskover

The firstindextime field supports the full range of Lucene date query syntax. Use these searches in the Diskover web interface search bar.

Basic Searches

Relative Date Searches

Diskover supports relative date math using now as the anchor point:

Date Range Searches

Date Math Reference

Diskover uses Lucene date math syntax:

Combined Searches

Combine first index time with other fields for powerful queries:

Backup Automation Example

To identify files for incremental backup (arrived in the last 7 days in a specific folder):

parent_path:/mnt/data/production/* AND firstindextime:[now-7d TO now]

Storage Growth Analytics Example

To analyze data arrival by quarter for capacity planning:

firstindextime:[2025-01-01 TO 2025-03-31]   # Q1 arrivals
firstindextime:[2025-04-01 TO 2025-06-30]   # Q2 arrivals
firstindextime:[2025-07-01 TO 2025-09-30]   # Q3 arrivals
firstindextime:[2025-10-01 TO 2025-12-31]   # Q4 arrivals

Troubleshooting

Common Issues

Verifying Plugin Operation

1. Check if plugin is loading:

   Linux:

   grep -i "firstindextime" /var/log/diskover/diskover.log | tail -20

   Windows:
   Check Diskover service logs or configured log location.

2. Test plugin import manually:

   Linux:

   cd /opt/diskover
   python3 -c "from plugins.firstindextime import *; print('Plugin loaded successfully')"

   Windows:

   cd "C:\Program Files\Diskover"
   python -c "from plugins.firstindextime import *; print('Plugin loaded successfully')"

Cache Management

The plugin stores first index times in a SQLite cache. If you need to clear the cache (which will reset all first index times), use:

Linux:

rm -rf /opt/diskover/__diskover_firstindextime_plugin_cache__/

Windows:

Remove-Item -Recurse -Force "C:\Program Files\Diskover\__diskover_firstindextime_plugin_cache__"

Warning: Clearing the cache will cause all files to receive new first index times on the next scan. Only clear the cache if you intentionally want to reset arrival tracking.

Debug Logging

Enable verbose logging in the plugin configuration to troubleshoot issues:

{
  "verbose": true
}

Then monitor logs during indexing:

Linux:

tail -f /var/log/diskover/diskover.log | grep -i firstindextime

Support

• Diskover Documentation

• Diskover Support Portal

• ISO 8601 Date Format Reference

Last Updated: January 2026
Diskover Data, Inc.