BAM Info Index Plugin

License: PRO+ (Professional Edition or higher)
Plugin Type: Index Plugin
Author: Diskover Data, Inc.

Overview

The BAM Info plugin automatically extracts metadata from bioinformatics sequence alignment files during Diskover indexing. If your organization stores genomics data—such as DNA or RNA sequencing results—this plugin makes those files searchable by the software tools and processing parameters used to create them.

What does this plugin do?

When Diskover scans your storage, the BAM Info plugin reads the header information embedded in BAM, SAM, and CRAM files. These headers contain valuable metadata about:

• Which alignment programs processed the file (e.g., STAR, BWA, Bowtie2)

• Software version numbers used during processing

• Command-line parameters that were executed

• Custom annotations added by your bioinformatics pipelines

This metadata becomes searchable in Diskover, allowing you to find files based on how they were created—not just their names or locations.

Why is this useful?

Understanding BAM/SAM/CRAM Formats

If you're an IT administrator managing storage for a genomics team, you may encounter these file types without knowing exactly what they contain. This section provides the background you need to understand what the plugin extracts and why it matters.

What Are These Files?

BAM, SAM, and CRAM files store sequence alignment data—the results of mapping DNA or RNA sequencing reads to a reference genome. Think of them as the processed output of a sequencing experiment.

Why Do These Files Get So Large?

A single human genome sequencing run can produce BAM files ranging from 5 GB to over 100 GB. Organizations with active sequencing programs may have thousands of these files, consuming petabytes of storage. Understanding what's in these files helps you make informed decisions about storage tiering and data lifecycle management.

What's Inside These Files?

Every BAM/SAM/CRAM file has two main sections:

1. Header Section (what this plugin reads):

   • Reference genome information

   • Program (PG) records: Which software processed the file and version numbers

   • Comment (CO) records: Command-line parameters, custom annotations, pipeline identifiers

2. Alignment Records (the bulk of the file):

   • Millions of individual sequence reads and their positions

   • Quality scores and alignment details

The plugin extracts metadata from the header section only, which is fast and doesn't require reading the entire file.

Why Header Metadata Matters for Storage Management

Requirements

Python Dependencies

System Requirements

• Python 3.9 or higher

• Diskover indexer with plugin support enabled

• Read access to BAM/SAM/CRAM files during indexing

• Sufficient permissions to create cache directories

Optional: CRAM Format Support

CRAM files use reference-based compression, which may require additional system libraries:

Note: Most organizations primarily use BAM format. CRAM support is only needed if your teams have adopted CRAM for archival storage.

Installation

Step 1: Install Python Dependencies

Linux:

python3 -m pip install pysam==0.22.1

Windows:

python -m pip install pysam==0.22.1

Step 2: Verify Installation

Linux:

python3 -c "import pysam; print(f'pysam version: {pysam.__version__}')"

Windows:

python -c "import pysam; print(f'pysam version: {pysam.__version__}')"

Expected output:

pysam version: 0.22.1

Step 3: Configure the Plugin

1. Navigate to Diskover Admin > Plugins > Index Plugins > BAM Info

2. Enable the plugin and configure parameters as needed (see Configuration section below)

3. Save the configuration

Step 4: Enable in Index Task Configuration

1. Navigate to Diskover > Configurations > [Your Configuration Name]

2. Scroll to the bottom to find Index Plugins Enablement

3. Enable the BAM Info plugin

4. Save the configuration

The plugin will now run automatically during any scan that uses this configuration.

Configuration

The BAM Info plugin configuration controls which metadata is extracted from alignment files.

Configuration Parameters

Understanding PG_program_ids

This setting determines which alignment programs are tracked. Only files processed by programs in this list will have their program metadata indexed.

Common alignment programs:

Tip: Ask your bioinformatics team which alignment tools they use, then add those program IDs to the configuration.

Understanding CO_list

BAM files often contain custom comments in KEY:VALUE format. This setting specifies which comment keys to extract.

Configuration Examples

Default Configuration (tracks STAR and BWA):

{
  "PG_enabled": true,
  "PG_program_ids": ["STAR", "bwa"],
  "CO_enabled": true,
  "CO_list": ["ANNID"],
  "extensions": {"sam": "", "bam": "b", "cram": "c"}
}

Expanded Configuration (tracks multiple alignment tools and custom fields):

{
  "PG_enabled": true,
  "PG_program_ids": ["STAR", "bwa", "minimap2", "bowtie2", "hisat2"],
  "CO_enabled": true,
  "CO_list": ["ANNID", "PIPELINE", "PROJECT", "SAMPLE_ID"],
  "extensions": {"sam": "", "bam": "b", "cram": "c"}
}

BAM-Only Configuration (skip SAM and CRAM files):

{
  "PG_enabled": true,
  "PG_program_ids": ["STAR", "bwa"],
  "CO_enabled": true,
  "CO_list": ["ANNID"],
  "extensions": {"bam": "b"}
}

Cache Configuration

The plugin caches extracted metadata to improve re-scan performance. Cache is stored in:

• Linux: /opt/diskover/__bam_plugin_cache__/

• Windows: C:\Program Files\Diskover\__bam_plugin_cache__\

The cache uses file modification time (mtime) to detect changes—if a file hasn't changed since the last scan, cached metadata is reused.

Indexed Fields / Elasticsearch Mappings

The BAM Info plugin adds a bam_info object to indexed documents containing metadata extracted from BAM file headers.

Field Mappings

Example Document

A BAM file processed by STAR version 2.7.10a with custom annotations would have metadata like this:

{
  "name": "sample_001_aligned.bam",
  "extension": "bam",
  "size": 15234567890,
  "bam_info": {
    "pg": [
      {
        "id": "STAR",
        "vn": "2.7.10a"
      }
    ],
    "co_cmd": "user command line: STAR --runThreadN 8 --genomeDir /ref/GRCh38 --readFilesIn sample_001_R1.fastq.gz sample_001_R2.fastq.gz --outSAMtype BAM SortedByCoordinate",
    "co_cmd_checksum": "a3f2b8c9d4e5f6a7b8c9d0e1f2a3b4c5",
    "co": [
      {
        "key": "ANNID",
        "value": "ANN001"
      },
      {
        "key": "PROJECT",
        "value": "PROJ_2024_001"
      }
    ]
  }
}

Searching in Diskover

Once the plugin indexes your BAM files, you can search for them using the metadata fields. Enter these queries in the Diskover search bar.

Program and Version Searches

Command Line and Annotation Searches

Tip: Use the command line checksum to find files processed with identical parameters—useful for verifying consistency across a project.

Combined and Storage Management Searches

Troubleshooting

Common Issues

Verifying Plugin Installation

Check if pysam is installed:

Linux:

python3 -c "import pysam; print(f'pysam version: {pysam.__version__}')"

Windows:

python -c "import pysam; print(f'pysam version: {pysam.__version__}')"

Test plugin import:

Linux:

cd /opt/diskover
python3 -c "from plugins.baminfo import *; print('Plugin loaded successfully')"

Windows:

cd "C:\Program Files\Diskover"
python -c "from plugins.baminfo import *; print('Plugin loaded successfully')"

Checking BAM File Headers

If metadata isn't being extracted, verify the file contains the expected headers:

# View all headers
samtools view -H /path/to/file.bam | head -50

# Check for Program Group headers
samtools view -H /path/to/file.bam | grep "^@PG"

# Check for Comment headers
samtools view -H /path/to/file.bam | grep "^@CO"

Cache Management

To clear the plugin cache and force re-extraction of all metadata:

Linux:

rm -rf /opt/diskover/__bam_plugin_cache__/

Windows:

rmdir /s /q "C:\Program Files\Diskover\__bam_plugin_cache__\"

Note: Clearing cache will cause the next scan to re-process all BAM files, which may increase scan time.

Debug Logging

To enable verbose logging for troubleshooting:

1. Enable verbose/debug logging in the BAM Info plugin configuration within Diskover Admin

2. Run a scan and monitor the logs

View logs:

Linux:

tail -f /var/log/diskover/diskover.log | grep baminfo

Look for these log messages:

Support

• Diskover Documentation

• Diskover Support Portal

• pysam Documentation

• samtools Documentation

Last Updated: January 2026
Diskover Data, Inc.