Grafana

License: PRO+ (Professional Edition or higher)
Plugin Type: Index Plugin
Author: Diskover Data, Inc.

Overview

The Grafana plugin enables time series analysis of your storage metrics by exporting directory-level data to a dedicated Elasticsearch index. While Diskover's primary indices provide point-in-time snapshots of your storage, this plugin creates a historical record that accumulates over time—allowing you to visualize storage growth trends, perform capacity planning, and answer questions like "How has this directory grown over the past six months?"

At the end of each crawl, the plugin automatically queries directory information from your Diskover index and exports it to a daily Elasticsearch index (using Logstash-compatible naming by default). This data can then be visualized in Grafana dashboards connected to Elasticsearch.

What This Plugin Does

• Creates timestamped documents containing directory size, file counts, and directory counts

• Exports data to daily Elasticsearch indices (e.g., logstash-2025.01.21)

• Automatically includes file age distribution data when fileages are enabled

• Runs at the end of each crawl with no manual intervention required

Use Cases

Storage Administrators & Capacity Planners

• Track storage growth rates for specific directories over weeks or months

• Predict when storage thresholds will be reached based on historical trends

• Identify which storage areas are growing fastest to prioritize tier migration

• Support procurement planning with data-driven budget forecasting

Project Managers & Department Leads

• Monitor how project directories grow throughout the project lifecycle

• Identify active versus dormant projects based on storage activity patterns

• Compare storage utilization across departments or teams

Finance & Compliance

• Generate historical reports on departmental storage consumption for chargebacks

• Track data accumulation patterns for compliance documentation

• Forecast future storage costs based on observed growth trends

Understanding Time Series Data for Storage

Why Time Series Analysis Matters

Traditional storage analysis shows you what exists right now. Time series analysis shows you how things change over time. This distinction is crucial for capacity planning because knowing that a directory is 500 GB tells you far less than knowing it grew from 200 GB to 500 GB in the past three months.

The Grafana plugin transforms Diskover's snapshot data into time series data by:

1. Timestamping – Adding @timestamp to each document for temporal ordering

2. Daily Indexing – Creating separate indices per day that accumulate over time

3. Metric Extraction – Pulling key metrics (size, counts) from directory documents

Why Separate Indices?

Diskover's primary indices are optimized for searching and analyzing current data. The Grafana plugin creates separate indices because each serves a different purpose:

How Data Flows

When a crawl completes, the plugin:

1. Queries all directories from the Diskover index where dir_depth is within your configured maximum

2. Extracts path, size, file count, directory count, and fileages (if available)

3. Adds a timestamp marking when the crawl completed

4. Uploads all documents to a daily index (e.g., logstash-2025.01.21)

If multiple crawls occur on the same day, documents are appended to the same daily index.

Grafana vs. Grafana Cloud Plugin

Diskover provides two plugins for time series visualization. Choose based on your infrastructure:

Use this plugin if you maintain an on-premise Elasticsearch cluster or have security requirements that mandate data stays within your network.

Requirements

System Requirements

• Python 3.9 or higher

• Diskover indexer with plugin support enabled

• Elasticsearch cluster (same cluster as primary Diskover index)

• Grafana instance with Elasticsearch datasource capability

• Network connectivity between Diskover and Elasticsearch

Python Dependencies

This plugin has no external Python dependencies beyond the Diskover core libraries and the Elasticsearch Python client (already included with Diskover).

Installation

Step 1: Install Dependencies

No additional installation is required. The plugin uses libraries already included with Diskover.

Step 2: Configure the Plugin

1. Navigate to Diskover Admin > Plugins > Index Plugins > Grafana

2. Enable the plugin and configure parameters as needed (see Configuration section below)

3. Save the configuration

Step 3: Enable in Index Task Configuration

1. Navigate to Diskover > Configurations > [Your Configuration Name]

2. At the bottom within Index Plugins Enablement, enable the Grafana plugin

3. The plugin will now run automatically at the end of each crawl using this configuration

Step 4: Enable Fileages (Optional but Recommended)

To include file age distribution data in your time series exports:

1. Navigate to Diskover > Configurations > [Your Configuration Name]

2. Locate the File Age Groups section

3. Enable and configure your desired age group thresholds

4. Save the configuration

When fileages are enabled, the Grafana plugin automatically includes this data in each export.

Configuration

Configuration Parameters

Understanding max_dir_depth

The max_dir_depth parameter controls which directories are exported to the time series index. This setting balances granularity against data volume.

Recommendation: Start with max_dir_depth: 2 and increase only if you need deeper granularity. Higher values significantly increase the number of documents generated per crawl.

Configuration Examples

Standard Configuration (Recommended)

Capture root and top two directory levels with Logstash-compatible naming:

{
  "max_dir_depth": 2,
  "index_prefix": "logstash-"
}

This creates indices like logstash-2025.01.21 containing directories at depth 0, 1, and 2.

Shallow Capture for Large Environments

For environments with many top-level directories, limit to the first level only to reduce data volume:

{
  "max_dir_depth": 1,
  "index_prefix": "logstash-"
}

Custom Index Prefix

Use a custom prefix to separate Diskover metrics from other Logstash data:

{
  "max_dir_depth": 2,
  "index_prefix": "diskover-metrics-"
}

This creates indices like diskover-metrics-2025.01.21.

Deep Capture for Project Hierarchies

For project-based storage with nested structures requiring granular tracking:

{
  "max_dir_depth": 4,
  "index_prefix": "logstash-"
}

Note: Deep captures can generate large numbers of documents. Monitor your index sizes when using higher depth values.

Indexed Fields / Elasticsearch Mappings

Index Naming Convention

The plugin creates indices using a daily naming pattern:

<index_prefix>YYYY.MM.DD

Examples:

• logstash-2025.01.21

• logstash-2025.01.22

• diskover-metrics-2025.01.21

If multiple crawls complete on the same day, documents are appended to the existing daily index.

Field Mappings

Each document in the time series index contains:

Fileages Subfields

When fileages are enabled in your Diskover configuration, the fileages object provides age distribution data:

Example Document

A document generated at the end of a crawl:

{
  "@timestamp": "2025-01-21T14:30:45.123456",
  "path": "/mnt/data/projects/alpha",
  "size": 10737418240,
  "size_du": 11274289152,
  "file_count": 15234,
  "dir_count": 342,
  "fileages": {
    "1d": 50,
    "7d": 200,
    "30d": 500,
    "90d": 1200,
    "180d": 2000,
    "365d": 5000,
    "older": 6284
  }
}

Grafana Setup

Configuring the Elasticsearch Datasource

To visualize the time series data in Grafana, add an Elasticsearch datasource:

1. In Grafana, navigate to Configuration > Data Sources > Add data source

2. Select Elasticsearch

3. Configure the connection:

   • URL: Your Elasticsearch URL (e.g., http://localhost:9200)

   • Index name: Use a pattern matching your index prefix (e.g., logstash-* or diskover-metrics-*)

   • Time field name: @timestamp

   • Version: Select your Elasticsearch version

4. Click Save & Test to verify the connection

Building Dashboards

With the datasource configured, you can create panels to visualize:

• Storage growth over time – Line charts showing size for specific paths

• File accumulation trends – Time series of file_count changes

• Age distribution changes – Stacked area charts using fileages data

• Comparative analysis – Multiple paths on a single chart for comparison

For detailed Grafana configuration and panel creation, refer to the Grafana Elasticsearch Datasource Documentation.

Example Queries

Track size of a specific directory over time:

• Query: path:"/mnt/data/projects/alpha"

• Metric: Average of size

• Group by: Date histogram on @timestamp

Compare growth across multiple directories:

• Query: path:"/mnt/data/projects/*"

• Metric: Average of size

• Group by: Terms on path, then date histogram on @timestamp

Monitor file count changes:

• Query: *

• Metric: Sum of file_count

• Group by: Date histogram on @timestamp

Troubleshooting

Common Issues

Verifying Plugin Activity

Check Diskover logs for plugin activity:

Linux:

grep -i "grafana" /var/log/diskover/diskover.log | tail -20

Windows:
Check Diskover service logs or your configured log location.

Verifying Index Creation

Check if the time series index was created:

curl -X GET "localhost:9200/_cat/indices/logstash-*?v"

Checking Document Count

Verify documents exist in the index:

curl -X GET "localhost:9200/logstash-*/_count" | jq '.count'

Debug Logging

To enable verbose logging for troubleshooting, adjust the logging level in your Diskover configuration. Plugin logs will appear with the prefix diskover.grafana_plugin.

Support

• Diskover Documentation

• Diskover Support Portal

• Grafana Documentation

• Grafana Elasticsearch Datasource Documentation

Last Updated: January 2026
Diskover Data, Inc.