Bioinformatics and Genomics File Formats Reference

This reference covers file formats used in genomics, transcriptomics, sequence analysis, and related bioinformatics applications.

Sequence Data Formats

.fasta / .fa / .fna - FASTA Format

Description: Text-based format for nucleotide or protein sequences Typical Data: DNA, RNA, or protein sequences with headers Use Cases: Sequence storage, BLAST searches, alignments Python Libraries: - Biopython: SeqIO.parse('file.fasta', 'fasta') - pyfaidx: Fast indexed FASTA access - screed: Fast sequence parsing EDA Approach: - Sequence count and length distribution - GC content analysis - N content (ambiguous bases) - Sequence ID parsing - Duplicate detection - Quality metrics for assemblies (N50, L50)

.fastq / .fq - FASTQ Format

Description: Sequence data with base quality scores Typical Data: Raw sequencing reads with Phred quality scores Use Cases: NGS data, quality control, read mapping Python Libraries: - Biopython: SeqIO.parse('file.fastq', 'fastq') - pysam: Fast FASTQ/BAM operations - HTSeq: Sequencing data analysis EDA Approach: - Read count and length distribution - Quality score distribution (per-base, per-read) - GC content and bias - Duplicate rate estimation - Adapter contamination detection - k-mer frequency analysis - Encoding format validation (Phred33/64)

.sam - Sequence Alignment/Map

Description: Tab-delimited text format for alignments Typical Data: Aligned sequencing reads with mapping quality Use Cases: Read alignment storage, variant calling Python Libraries: - pysam: pysam.AlignmentFile('file.sam', 'r') - HTSeq: HTSeq.SAM_Reader('file.sam') EDA Approach: - Mapping rate and quality distribution - Coverage analysis - Insert size distribution (paired-end) - Alignment flags distribution - CIGAR string patterns - Mismatch and indel rates - Duplicate and supplementary alignment counts

.bam - Binary Alignment/Map

Description: Compressed binary version of SAM Typical Data: Aligned reads in compressed format Use Cases: Efficient storage and processing of alignments Python Libraries: - pysam: Full BAM support with indexing - bamnostic: Pure Python BAM reader EDA Approach: - Same as SAM plus: - Compression ratio analysis - Index file (.bai) validation - Chromosome-wise statistics - Strand bias detection - Read group analysis

.cram - CRAM Format

Description: Highly compressed alignment format Typical Data: Reference-compressed aligned reads Use Cases: Long-term storage, space-efficient archives Python Libraries: - pysam: CRAM support (requires reference) - Reference genome must be accessible EDA Approach: - Compression efficiency vs BAM - Reference dependency validation - Lossy vs lossless compression assessment - Decompression performance - Similar alignment metrics as BAM

.bed - Browser Extensible Data

Description: Tab-delimited format for genomic features Typical Data: Genomic intervals (chr, start, end) with annotations Use Cases: Peak calling, variant annotation, genome browsing Python Libraries: - pybedtools: pybedtools.BedTool('file.bed') - pyranges: pyranges.read_bed('file.bed') - pandas: Simple BED reading EDA Approach: - Feature count and size distribution - Chromosome distribution - Strand bias - Score distribution (if present) - Overlap and proximity analysis - Coverage statistics - Gap analysis between features

.bedGraph - BED with Graph Data

Description: BED format with per-base signal values Typical Data: Continuous-valued genomic data (coverage, signals) Use Cases: Coverage tracks, ChIP-seq signals, methylation Python Libraries: - pyBigWig: Can convert to bigWig - pybedtools: BedGraph operations EDA Approach: - Signal distribution statistics - Genome coverage percentage - Signal dynamics (peaks, valleys) - Chromosome-wise signal patterns - Quantile analysis - Zero-coverage regions

.bigWig / .bw - Binary BigWig

Description: Indexed binary format for genome-wide signal data Typical Data: Continuous genomic signals (compressed and indexed) Use Cases: Efficient genome browser tracks, large-scale data Python Libraries: - pyBigWig: pyBigWig.open('file.bw') - pybbi: BigWig/BigBed interface EDA Approach: - Signal statistics extraction - Zoom level analysis - Regional signal extraction - Efficient genome-wide summaries - Compression efficiency - Index structure analysis

.bigBed / .bb - Binary BigBed

Description: Indexed binary BED format Typical Data: Genomic features (compressed and indexed) Use Cases: Large feature sets, genome browsers Python Libraries: - pybbi: BigBed reading - pybigtools: Modern BigBed interface EDA Approach: - Feature density analysis - Efficient interval queries - Zoom level validation - Index performance metrics - Feature size statistics

.gff / .gff3 - General Feature Format

Description: Tab-delimited format for genomic annotations Typical Data: Gene models, transcripts, exons, regulatory elements Use Cases: Genome annotation, gene prediction Python Libraries: - BCBio.GFF: Biopython GFF module - gffutils: gffutils.create_db('file.gff3') - pyranges: GFF support EDA Approach: - Feature type distribution (gene, exon, CDS, etc.) - Gene structure validation - Strand balance - Hierarchical relationship validation - Phase validation for CDS - Attribute completeness - Gene model statistics (introns, exons per gene)

.gtf - Gene Transfer Format

Description: GFF2-based format for gene annotations Typical Data: Gene and transcript annotations Use Cases: RNA-seq analysis, gene quantification Python Libraries: - pyranges: pyranges.read_gtf('file.gtf') - gffutils: GTF database creation - HTSeq: GTF reading for counts EDA Approach: - Transcript isoform analysis - Gene structure completeness - Exon number distribution - Transcript length distribution - TSS and TES analysis - Biotype distribution - Overlapping gene detection

.vcf - Variant Call Format

Description: Text format for genetic variants Typical Data: SNPs, indels, structural variants with annotations Use Cases: Variant calling, population genetics, GWAS Python Libraries: - pysam: pysam.VariantFile('file.vcf') - cyvcf2: Fast VCF parsing - PyVCF: Older but comprehensive EDA Approach: - Variant count by type (SNP, indel, SV) - Quality score distribution - Allele frequency spectrum - Transition/transversion ratio - Heterozygosity rates - Missing genotype analysis - Hardy-Weinberg equilibrium - Annotation completeness (if annotated)

.bcf - Binary VCF

Description: Compressed binary variant format Typical Data: Same as VCF but binary Use Cases: Efficient variant storage and processing Python Libraries: - pysam: Full BCF support - cyvcf2: Optimized BCF reading EDA Approach: - Same as VCF plus: - Compression efficiency - Indexing validation - Read performance metrics

.gvcf - Genomic VCF

Description: VCF with reference confidence blocks Typical Data: All positions (variant and non-variant) Use Cases: Joint genotyping workflows, GATK Python Libraries: - pysam: GVCF support - Standard VCF parsers EDA Approach: - Reference block analysis - Coverage uniformity - Variant density - Genotype quality across genome - Reference confidence distribution

RNA-Seq and Expression Data

.counts - Gene Count Matrix

Description: Tab-delimited gene expression counts Typical Data: Gene IDs with read counts per sample Use Cases: RNA-seq quantification, differential expression Python Libraries: - pandas: pd.read_csv('file.counts', sep='\t') - scanpy (for single-cell): sc.read_csv() EDA Approach: - Library size distribution - Detection rate (genes per sample) - Zero-inflation analysis - Count distribution (log scale) - Outlier sample detection - Correlation between replicates - PCA for sample relationships

.tpm / .fpkm - Normalized Expression

Description: Normalized gene expression values Typical Data: TPM (transcripts per million) or FPKM values Use Cases: Cross-sample comparison, visualization Python Libraries: - pandas: Standard CSV reading - anndata: For integrated analysis EDA Approach: - Expression distribution - Highly expressed gene identification - Sample clustering - Batch effect detection - Coefficient of variation analysis - Dynamic range assessment

.mtx - Matrix Market Format

Description: Sparse matrix format (common in single-cell) Typical Data: Sparse count matrices (cells × genes) Use Cases: Single-cell RNA-seq, large sparse matrices Python Libraries: - scipy.io: scipy.io.mmread('file.mtx') - scanpy: sc.read_mtx('file.mtx') EDA Approach: - Sparsity analysis - Cell and gene filtering thresholds - Doublet detection metrics - Mitochondrial fraction - UMI count distribution - Gene detection per cell

.h5ad - Anndata Format

Description: HDF5-based annotated data matrix Typical Data: Expression matrix with metadata (cells, genes) Use Cases: Single-cell RNA-seq analysis with Scanpy Python Libraries: - scanpy: sc.read_h5ad('file.h5ad') - anndata: Direct AnnData manipulation EDA Approach: - Cell and gene counts - Metadata completeness - Layer availability (raw, normalized) - Embedding presence (PCA, UMAP) - QC metrics distribution - Batch information - Cell type annotation coverage

.loom - Loom Format

Description: HDF5-based format for omics data Typical Data: Expression matrices with metadata Use Cases: Single-cell data, RNA velocity analysis Python Libraries: - loompy: loompy.connect('file.loom') - scanpy: Can import loom files EDA Approach: - Layer analysis (spliced, unspliced) - Row and column attribute exploration - Graph connectivity analysis - Cluster assignments - Velocity-specific metrics

.rds - R Data Serialization

Description: R object storage (often Seurat objects) Typical Data: R analysis results, especially single-cell Use Cases: R-Python data exchange Python Libraries: - pyreadr: pyreadr.read_r('file.rds') - rpy2: For full R integration - Conversion tools to AnnData EDA Approach: - Object type identification - Data structure exploration - Metadata extraction - Conversion validation

Alignment and Assembly Formats

.maf - Multiple Alignment Format

Description: Text format for multiple sequence alignments Typical Data: Genome-wide or local multiple alignments Use Cases: Comparative genomics, conservation analysis Python Libraries: - Biopython: AlignIO.parse('file.maf', 'maf') - bx-python: MAF-specific tools EDA Approach: - Alignment block statistics - Species coverage - Gap analysis - Conservation scoring - Alignment quality metrics - Block length distribution

.axt - Pairwise Alignment Format

Description: Pairwise alignment format (UCSC) Typical Data: Pairwise genomic alignments Use Cases: Genome comparison, synteny analysis Python Libraries: - Custom parsers (simple format) - bx-python: AXT support EDA Approach: - Alignment score distribution - Identity percentage - Syntenic block identification - Gap size analysis - Coverage statistics

.chain - Chain Alignment Format

Description: Genome coordinate mapping chains Typical Data: Coordinate transformations between genome builds Use Cases: Liftover, coordinate conversion Python Libraries: - pyliftover: Chain file usage - Custom parsers for chain format EDA Approach: - Chain score distribution - Coverage of source genome - Gap analysis - Inversion detection - Mapping quality assessment

.psl - Pattern Space Layout

Description: BLAT/BLAST alignment format Typical Data: Alignment results from BLAT Use Cases: Transcript mapping, similarity searches Python Libraries: - Custom parsers (tab-delimited) - pybedtools: Can handle PSL EDA Approach: - Match percentage distribution - Gap statistics - Query coverage - Multiple mapping analysis - Alignment quality metrics

Genome Assembly and Annotation

.agp - Assembly Golden Path

Description: Assembly structure description Typical Data: Scaffold composition, gap information Use Cases: Genome assembly representation Python Libraries: - Custom parsers (simple tab-delimited) - Assembly analysis tools EDA Approach: - Scaffold statistics (N50, L50) - Gap type and size distribution - Component length analysis - Assembly contiguity metrics - Unplaced contig analysis

.scaffolds / .contigs - Assembly Sequences

Description: Assembled sequences (usually FASTA) Typical Data: Assembled genomic sequences Use Cases: Genome assembly output Python Libraries: - Same as FASTA format - Assembly-specific tools (QUAST) EDA Approach: - Assembly statistics (N50, N90, etc.) - Length distribution - Coverage analysis - Gap (N) content - Duplication assessment - BUSCO completeness (if annotations available)

.2bit - Compressed Genome Format

Description: UCSC compact genome format Typical Data: Reference genomes (highly compressed) Use Cases: Efficient genome storage and access Python Libraries: - py2bit: py2bit.open('file.2bit') - twobitreader: Alternative reader EDA Approach: - Compression efficiency - Random access performance - Sequence extraction validation - Masked region analysis - N content and distribution

.sizes - Chromosome Sizes

Description: Simple format with chromosome lengths Typical Data: Tab-delimited chromosome names and sizes Use Cases: Genome browsers, coordinate validation Python Libraries: - Simple file reading with pandas - Built into many genomic tools EDA Approach: - Genome size calculation - Chromosome count - Size distribution - Karyotype validation - Completeness check against reference

Phylogenetics and Evolution

.nwk / .newick - Newick Tree Format

Description: Parenthetical tree representation Typical Data: Phylogenetic trees with branch lengths Use Cases: Evolutionary analysis, tree visualization Python Libraries: - Biopython: Phylo.read('file.nwk', 'newick') - ete3: ete3.Tree('file.nwk') - dendropy: Phylogenetic computing EDA Approach: - Tree structure analysis (tips, internal nodes) - Branch length distribution - Tree balance metrics - Ultrametricity check - Bootstrap support analysis - Topology validation

.nexus - Nexus Format

Description: Rich format for phylogenetic data Typical Data: Alignments, trees, character matrices Use Cases: Phylogenetic software interchange Python Libraries: - Biopython: Nexus support - dendropy: Comprehensive Nexus handling EDA Approach: - Data block analysis - Character type distribution - Tree block validation - Taxa consistency - Command block parsing - Format compliance checking

.phylip - PHYLIP Format

Description: Sequence alignment format (strict/relaxed) Typical Data: Multiple sequence alignments Use Cases: Phylogenetic analysis input Python Libraries: - Biopython: AlignIO.read('file.phy', 'phylip') - dendropy: PHYLIP support EDA Approach: - Alignment dimensions - Sequence length uniformity - Gap position analysis - Informative site calculation - Format variant detection (strict vs relaxed)

.paml - PAML Output

Description: Output from PAML phylogenetic software Typical Data: Evolutionary model results, dN/dS ratios Use Cases: Molecular evolution analysis Python Libraries: - Custom parsers for specific PAML programs - Biopython: Basic PAML parsing EDA Approach: - Model parameter extraction - Likelihood values - dN/dS ratio distribution - Branch-specific results - Convergence assessment

Protein and Structure Data

.embl - EMBL Format

Description: Rich sequence annotation format Typical Data: Sequences with extensive annotations Use Cases: Sequence databases, genome records Python Libraries: - Biopython: SeqIO.read('file.embl', 'embl') EDA Approach: - Feature annotation completeness - Sequence length and type - Reference information - Cross-reference validation - Feature overlap analysis

.genbank / .gb / .gbk - GenBank Format

Description: NCBI's sequence annotation format Typical Data: Annotated sequences with features Use Cases: Sequence databases, annotation transfer Python Libraries: - Biopython: SeqIO.parse('file.gb', 'genbank') EDA Approach: - Feature type distribution - CDS analysis (start codons, stops) - Translation validation - Annotation completeness - Source organism extraction - Reference and publication info - Locus tag consistency

.sff - Standard Flowgram Format

Description: 454/Roche sequencing data format Typical Data: Raw pyrosequencing flowgrams Use Cases: Legacy 454 sequencing data Python Libraries: - Biopython: SeqIO.parse('file.sff', 'sff') - Platform-specific tools EDA Approach: - Read count and length - Flowgram signal quality - Key sequence detection - Adapter trimming validation - Quality score distribution

.hdf5 (Genomics Specific)

Description: HDF5 for genomics (10X, Hi-C, etc.) Typical Data: High-throughput genomics data Use Cases: 10X Genomics, spatial transcriptomics Python Libraries: - h5py: Low-level access - scanpy: For 10X data - cooler: For Hi-C data EDA Approach: - Dataset structure exploration - Barcode statistics - UMI counting - Feature-barcode matrix analysis - Spatial coordinates (if applicable)

.cool / .mcool - Cooler Format

Description: HDF5-based Hi-C contact matrices Typical Data: Chromatin interaction matrices Use Cases: 3D genome analysis, Hi-C data Python Libraries: - cooler: cooler.Cooler('file.cool') - hicstraw: For .hic format EDA Approach: - Resolution analysis - Contact matrix statistics - Distance decay curves - Compartment analysis - TAD boundary detection - Balance factor validation

.hic - Hi-C Binary Format

Description: Juicer binary Hi-C format Typical Data: Multi-resolution Hi-C matrices Use Cases: Hi-C analysis with Juicer tools Python Libraries: - hicstraw: hicstraw.HiCFile('file.hic') - straw: C++ library with Python bindings EDA Approach: - Available resolutions - Normalization methods - Contact statistics - Chromosomal interactions - Quality metrics

.bw (ChIP-seq / ATAC-seq specific)

Description: BigWig files for epigenomics Typical Data: Coverage or enrichment signals Use Cases: ChIP-seq, ATAC-seq, DNase-seq Python Libraries: - pyBigWig: Standard bigWig access EDA Approach: - Peak enrichment patterns - Background signal analysis - Sample correlation - Signal-to-noise ratio - Library complexity metrics

.narrowPeak / .broadPeak - ENCODE Peak Formats

Description: BED-based formats for peaks Typical Data: Peak calls with scores and p-values Use Cases: ChIP-seq peak calling output Python Libraries: - pybedtools: BED-compatible - Custom parsers for peak-specific fields EDA Approach: - Peak count and width distribution - Signal value distribution - Q-value and p-value analysis - Peak summit analysis - Overlap with known features - Motif enrichment preparation

.wig - Wiggle Format

Description: Dense continuous genomic data Typical Data: Coverage or signal tracks Use Cases: Genome browser visualization Python Libraries: - pyBigWig: Can convert to bigWig - Custom parsers for wiggle format EDA Approach: - Signal statistics - Coverage metrics - Format variant (fixedStep vs variableStep) - Span parameter analysis - Conversion efficiency to bigWig

.ab1 - Sanger Sequencing Trace

Description: Binary chromatogram format Typical Data: Sanger sequencing traces Use Cases: Capillary sequencing validation Python Libraries: - Biopython: SeqIO.read('file.ab1', 'abi') - tracy tools: For quality assessment EDA Approach: - Base calling quality - Trace quality scores - Mixed base detection - Primer and vector detection - Read length and quality region - Heterozygosity detection

.scf - Standard Chromatogram Format

Description: Sanger sequencing chromatogram Typical Data: Base calls and confidence values Use Cases: Sequencing trace analysis Python Libraries: - Biopython: SCF format support EDA Approach: - Similar to AB1 format - Quality score profiles - Peak height ratios - Signal-to-noise metrics

.idx - Index Files (Generic)

Description: Index files for various formats Typical Data: Fast random access indices Use Cases: Efficient data access (BAM, VCF, etc.) Python Libraries: - Format-specific libraries handle indices - pysam: Auto-handles BAI, CSI indices EDA Approach: - Index completeness validation - Binning strategy analysis - Access performance metrics - Index size vs data size ratio