Diskover Sizing Guide

How to use this guide

• Part 1 (Sizing Guide for Endusers) answers one question: for this many files, what resources does Diskover need? It takes four questions and a lookup table to get an answer. No Elasticsearch/Opensearch knowledge is required.

• Part 2 (Technical Background) explains how the Part 1 numbers were derived. It's for engineers and architects who want the detail.

Part 1: Diskover Sizing Guide

Step 1: Answer four questions

1. How many files and folders will Diskover scan?

This is the total count of files and folders across all storage Diskover will scan. Sizing is based on the number of files, not terabytes. For example, 500 TB of large video files can need a smaller system than 50 TB of millions of small documents.

2. How often will the files be scanned, and how much history will be kept?

Each scan is saved as a snapshot of the storage at that point in time. Keeping more snapshots allows for comparison and trend over time, but every snapshot kept adds disk space. The two standard options are (but not limited to):

• Weekly scans, 6 months of history (24 snapshots kept). This is the standard setup.

• Nightly scans, 2 weeks of history (14 snapshots kept).

If unsure, a great starting point is weekly scans with 6 months of history. This can always be updated.

3. How many top-level paths (shares, volumes, or buckets) will be scanned?

Most users fall well within standard sizing. A very high number of top paths combined with long history can require extra servers. See Step 3.

4. Is there any scan history that must not be lost?

Diskover can always rebuild its current view of the storage by rescanning, so most users don't need a backup copy of their Diskover data. Past snapshots can't be recreated, though. Once the storage changes, the history of what it used to look like is gone.

Ask:

• Will any storage be migrated, decommissioned, or deleted, making the Diskover scan the only record of what was there?

• Will point-in-time history for compliance, audits, legal hold, or chargeback reporting be needed?

If neither applies, the answer is no, which is the standard configuration.

If either applies, then Diskover needs a second copy of the data, called a replica. A replica doubles the disk space from Step 2. Replicas also require multiple Elasticsearch/Opensearch nodes, so the single-server option isn't compatible with a need for replicas.

Step 2: Find the recommended configuration

A standard Diskover deployment has three types of servers:

• 1 web host: runs the Diskover web interface and supporting services

• 3 or more Elasticsearch/Opensearch servers: store and search the scan data

• task worker(s): runs the scans - number of workers is dependent on use case

Servers can be physical or virtual, on-premises or cloud.

Server counts and specs

Webhost:

Task Worker(s):

Elasticsearch/Opensearch:

• Figures assume the top of each file count range. Smaller customers in a range need less. Use the quick math below.

• If the Diskover needs a replica (Step 1, question 4), double these disk space figures for Elasticsearch/Opensearch.

• These are minimums. Elasticsearch/Opensearch stops accepting new data when a disk gets close to full (around 85–95%), so plan disks with room to spare and room for growth.

Quick math for any Diskover instance

• About 1 GB of disk for every 5 million files, for each snapshot kept.

• Total disk = (Files ÷ 5 million) × Snapshots kept (× 2 if a replica is needed)

• Divide the total by the number of Elasticsearch/Opensearch servers to get disk per server.

Example: 400 million files, weekly scans, 6 months of history (24 snapshots), no replica: 400 ÷ 5 = 80 GB per snapshot × 24 = 1,920 GB total ÷ 4 servers (Large) = 480 GB per Elasticsearch/Opensearch server

Small environments: single-server option

For Diskover instances scanning under 100 million files in special circumstances, such as a small environment, an evaluation, or limited infrastructure, all Diskover components can run on one server:

Disk figures include about 100 GB for the operating system and Diskover application. Trade-offs: no redundancy, replicas aren't possible, and all components share one server's resources.

Step 3: Check the number of top-level paths

Each top-level path creates its own set of data for every snapshot kept, and each Elasticsearch/Opensearch server can only hold so many. For most Diskover instances this never matters. For Diskover instances with a lot of top paths and long history, it can require more Elasticsearch/Opensearch servers than the file count alone suggests.

On a standard 3-server system:

• Weekly scans, 6 months of history: more than about 85 top paths needs additional Elasticsearch/Opensearch servers

• Nightly scans, 2 weeks of history: more than about 150 top paths needs additional Elasticsearch/Opensearch servers

Rule of thumb: each Elasticsearch/Opensearch server supports about 700 (top paths × snapshots kept).

Example: 200 top paths × 24 snapshots = 4,800 ÷ 700 = 7 Elasticsearch/Opensearch servers

Use whichever is larger: the server count from Step 2 or from Step 3.

Glossary

Part 2: Technical Background

This section explains the assumptions and calculations behind Part 1.

2.1 Assumptions

• Index size: about 1 GB per 5 million files/folders (about 20 GB per 100 million) for standard metadata scans.

• Replicas: 0 by default.

• Primary shards: 1 per index.

• Standard layout: 1 web host, 3 or more Elasticsearch/Opensearch data nodes, 1 task worker.

• Default retention: weekly scans with 24 indices per top path (6 months), or nightly scans with 14 indices per top path (2 weeks).

• Disk figures are calculated minimums for index data, rounded up to the nearest 50 GB. They do not include headroom for disk watermarks (see 2.5) or growth.

2.2 Total Index Storage Formula

Total Storage (GB) = (Total Files ÷ 5,000,000) × Retained Indices × (1 + Replicas)

• Total Files: total files and folders across all scanned top paths

• Retained Indices: snapshots kept per top path (use 1 if only the latest scan is kept)

• Replicas: replica count per index (default 0)

Disk per Elasticsearch/Opensearch node = Total Storage ÷ number of data nodes.

Example: 1 billion files, weekly scans, 24 retained, 0 replicas: (1,000M ÷ 5M) × 24 × 1 = 4,800 GB total, or 1,200 GB per node across 4 nodes. With 1 replica: 9,600 GB total, or 2,400 GB per node.

2.3 Replica Guidance

Replicas are not required by default. Diskover indices hold metadata that can be rebuilt by rescanning the source storage, so the scan process gives a degree of redundancy on its own. Replicas double (or more) the storage footprint and add indexing overhead during scans.

Recommend a replica when: a scan can't be rescanned and keeping that point-in-time snapshot of the storage is important. Examples include storage being migrated, decommissioned, or deleted, and history needed for compliance, audit, or legal purposes.

Search performance: Replicas can also improve search throughput. A search only needs one copy of each shard, so with replicas Elasticsearch/Opensearch can spread concurrent queries across more nodes, and adaptive replica selection routes each request to the least-busy copy. This helps most when many users or API clients query at once, or when searches run during active scans. Replicas do not make a single query faster, and on resource-constrained nodes extra replicas can hurt performance by competing for memory and disk. Most Diskover deployments have modest query concurrency, so search performance alone rarely justifies doubling storage.

Notes:

• Replicas require 2 or more data nodes. On single-server deployments, set replicas to 0, or the cluster will report yellow health.

• Replica count is a per-index setting (number_of_replicas) and can be changed on a live index. That allows replicas on only the indices that need protection rather than cluster-wide.

2.4 Shard Count Check

Total Shards = Top Paths × Retained Indices × Primary Shards per Index × (1 + Replicas)

Elasticsearch/Opensearch's default limit is 1,000 shards per data node (cluster.max_shards_per_node). Part 1 uses 700 per node (about 70% of the limit) as the planning threshold to leave room for growth and new top paths.

Example: 150 top paths × 24 retained × 1 primary × (1 + 0) = 3,600 shards. That exceeds a 3-node cluster's 2,100-shard planning threshold and calls for about 6 data nodes, even if total storage is small.

Options for high shard counts:

• Shorten retention for low-priority top paths

• Consolidate many small top paths by scanning at a higher-level parent path

• Confirm small indices use 1 primary shard

• Add data nodes

• Raise cluster.max_shards_per_node (this adds heap and cluster-state pressure, so do it cautiously and monitor)

2.5 Disk Watermarks

By default, Elasticsearch/Opensearch:

• stops allocating new shards to a node at 85% disk usage (low watermark)

• relocates shards off a node at 90% (high watermark)

• sets indices to read-only at 95% (flood stage)

Part 1 sizing does not include headroom for these thresholds. Usable capacity is below raw capacity, so plan disk monitoring and alerting to match.

2.6 Shard Sizing

• Target shard size: 20 GB – 40 GB (10 GB – 50 GB operational window).

• At about 1 GB per 5 million files, a 50 GB shard holds about 250 million files. Most Diskover indices fit in a single primary shard. Very large individual top paths may need more primary shards.

• The Lucene limit of about 2 billion documents per shard isn't a practical constraint at these shard sizes.

2.7 JVM Heap

• Allocate 50% of RAM to the Elasticsearch/Opensearch Java heap, up to a maximum of 31 GB. Staying at or below 31 GB preserves compressed object pointers.

• The remaining RAM is used for OS page cache, Lucene segment caches, and thread stacks.

• 32 GB RAM node → 16 GB heap. 64 GB RAM node → 31 GB heap.

2.8 Component Specifications

Task workers: Scale core count and worker count with scan volume. Deploy additional workers near remote storage locations or object stores to avoid WAN bottlenecks.

2.9 Network

• 1 GbE minimum. 10 GbE is recommended for multi-node clusters and at 1.5 billion files and above.

• Scanning is mostly metadata operations and sensitive to latency, so place task workers close to the storage they scan.

2.10 Scaling Beyond 2 Billion Files

For systems over 2 billion files, scale out Elasticsearch/Opensearch:

• Start with about 1 additional data node per 500 million files beyond 2 billion, at the same 16-core / 64 GB spec.

• Recalculate disk with the formula in 2.2.

• Run the shard count check in 2.4.

2.11 Scope Notes

• Part 1 sizing assumes standard metadata scans. Plugins that read file contents, such as file hashing or content indexing, add read load on the source storage and lengthen scan times. Plan additional task worker capacity for those deployments.

• The index size estimate (1 GB per 5 million files) varies with path length, directory depth, and tagging. Validate against actual index sizes where possible.