BEDOPS: the fast, highly scalable and easily-parallelizable genome analysis toolkit

BEDOPS is an open-source command-line toolkit that performs highly efficient and scalable Boolean and other set operations, statistical calculations, archiving, conversion and other management of genomic data of arbitrary scale. Tasks can be easily split by chromosome for distributing whole-genome analyses across a computational cluster.

You can read more about BEDOPS and how it can be useful for your research in the Overview documentation, as well as in the original manuscript.

bedops - apply set operations on any number of BED inputs bedextract - efficiently extract BED features closest-features - matches nearest features between BED files	bedmap - map overlapping BED elements onto target regions, and optionally compute any number of common statistical operations	sort-bed - apply lexicographical sort to BED data starch and unstarch - compress and extract BED data starchcat - merge compressed archives Conversion tools - convert common genomic formats to BED
Set operations with bedops Compression characteristics of starch Independent testing	How to install BEDOPS Usage examples of BEDOPS tools in action BEDOPS user forum BEDOPS discusssion mailing list	Table summary of BEDOPS toolkit Starch v2 format specification Revision history Github repository

Citation

If you use BEDOPS in your research, please cite the following manuscript:

Shane Neph, M. Scott Kuehn, Alex P. Reynolds, et al. BEDOPS: high-performance genomic feature operations. Bioinformatics (2012) 28 (14): 1919-1920. doi: 10.1093/bioinformatics/bts277

Table of contents

• 1. Overview
  • 1.1. About BEDOPS
  • 1.2. Why you should use BEDOPS
    • 1.2.1. BEDOPS tools are flexible
    • 1.2.2. BEDOPS tools are fast and efficient
    • 1.2.3. BEDOPS tools make your work embarrassingly easy to parallelize
    • 1.2.4. BEDOPS tools are open, documented and supported
• 2. Installation
  • 2.1. Via pre-built packages
    • 2.1.1. Linux
    • 2.1.2. Mac OS X
  • 2.2. Via source code
  • 2.3. Building an OS X installer package for redistribution
• 3. Revision history
  • 3.1. Current version
    • 3.1.1. v2.3.0
  • 3.2. Previous versions
    • 3.2.1. v2.2.0b
    • 3.2.2. v2.2.0
    • 3.2.3. v2.1.1
    • 3.2.4. v2.1.0
    • 3.2.5. v2.0.0b
    • 3.2.6. v2.0.0a
    • 3.2.7. v1.2.5b
    • 3.2.8. v1.2.5
    • 3.2.9. v1.2.3
• 4. Usage examples
  • 4.1. Visualizing the relationship of SNPs and generic genomic features
    • 4.1.1. BEDOPS tools in use
    • 4.1.2. Script
    • 4.1.3. Discussion
    • 4.1.4. Downloads
  • 4.2. Collapsing multiple BED files into a master list by signal
    • 4.2.1. BEDOPS tools in use
    • 4.2.2. Script
    • 4.2.3. Discussion
  • 4.3. Measuring the frequency of signed distances between SNPs and nearest DHSes
    • 4.3.1. BEDOPS tools in use
    • 4.3.2. Script
    • 4.3.3. Discussion
    • 4.3.4. Downloads
  • 4.4. Finding the subset of SNPs within DHSes
    • 4.4.1. BEDOPS tools in use
    • 4.4.2. Script
    • 4.4.3. Discussion
    • 4.4.4. Downloads
  • 4.5. Smoothing raw tag count data across the genome
    • 4.5.1. BEDOPS tools in use
    • 4.5.2. Script
  • 4.6. Efficiently creating Starch-formatted archives with a cluster
    • 4.6.1. BEDOPS tools in use
    • 4.6.2. Script
    • 4.6.3. Discussion
      • 4.6.3.1. Splitting BED files
      • 4.6.3.2. Compressing BED subsets
      • 4.6.3.3. Stitching together compressed sets
  • 4.7. Working with many input files at once with bedops and bedmap
    • 4.7.1. Discussion
• 5. Performance
  • 5.1. Test environment and data
  • 5.2. Set operations with bedops
    • 5.2.1. Direct merge (sorted)
    • 5.2.2. Complement and intersection
    • 5.2.3. Direct merge (unsorted)
    • 5.2.4. Discussion
  • 5.3. Compression characteristics of starch
    • 5.3.1. Compression efficiency
    • 5.3.2. Extraction time
  • 5.4. Independent testing
    • 5.4.1. Genomic Region Operation Kit (GROK)
  • 5.5. Worst-case memory performance
• 6. Reference
  • 6.1. Set operations
    • 6.1.1. bedops
      • 6.1.1.1. Inputs and outputs
        • 6.1.1.1.1. Input
        • 6.1.1.1.2. Output
      • 6.1.1.2. Usage
      • 6.1.1.3. Operations
        • 6.1.1.3.1. Everything (-u, –everything)
        • 6.1.1.3.2. Element-of (-e, –element-of)
        • 6.1.1.3.3. Not-element-of (-n, –not-element-of)
        • 6.1.1.3.4. Complement (-c, –complement)
        • 6.1.1.3.5. Difference (-d, –difference)
        • 6.1.1.3.6. Symmetric difference (-s, –symmdiff)
        • 6.1.1.3.7. Intersect (-i, –intersect)
        • 6.1.1.3.8. Merge (-m, –merge)
        • 6.1.1.3.9. Partition (-p, –partition)
        • 6.1.1.3.10. Per-chromosome operations (–chrom)
        • 6.1.1.3.11. Range (–range)
      • 6.1.1.4. Starch support
      • 6.1.1.5. Error checking (–ec)
      • 6.1.1.6. Tips
        • 6.1.1.6.1. Chaining operations
        • 6.1.1.6.2. Sorting inputs
    • 6.1.2. bedextract
      • 6.1.2.1. How it works
      • 6.1.2.2. Inputs and outputs
        • 6.1.2.2.1. Input
        • 6.1.2.2.2. Output
      • 6.1.2.3. Usage
        • 6.1.2.3.1. Listing chromosomes
        • 6.1.2.3.2. Retrieving elements from a specific chromosome
        • 6.1.2.3.3. Retrieving elements which overlap target elements
          • 6.1.2.3.3.1. What are nested elements?
          • 6.1.2.3.3.2. Demonstration
      • 6.1.2.4. Downloads
    • 6.1.3. closest-features
      • 6.1.3.1. Inputs and outputs
        • 6.1.3.1.1. Input
        • 6.1.3.1.2. Output
      • 6.1.3.2. Usage
      • 6.1.3.3. Per-chromosome operations (–chrom)
      • 6.1.3.4. Error checking
      • 6.1.3.5. Downloads
  • 6.2. Statistics
    • 6.2.1. bedmap
      • 6.2.1.1. Inputs and outputs
        • 6.2.1.1.1. Input
        • 6.2.1.1.2. Output
      • 6.2.1.2. Usage
      • 6.2.1.3. Operations
        • 6.2.1.3.1. Overlap criteria
          • 6.2.1.3.1.1. Using --faster with --bp-ovr and --range
        • 6.2.1.3.2. Score operations
        • 6.2.1.3.3. Non-score operations
          • 6.2.1.3.3.1. Echo
          • 6.2.1.3.3.2. Element and overlap statistics
          • 6.2.1.3.3.3. Indicator
        • 6.2.1.3.4. Modifiers
          • 6.2.1.3.4.1. Range
          • 6.2.1.3.4.2. Using --faster with --range
          • 6.2.1.3.4.3. Formatting score output
          • 6.2.1.3.4.4. Delimiters
      • 6.2.1.4. Per-chromosome operations (–chrom)
      • 6.2.1.5. Starch support
      • 6.2.1.6. Error checking
      • 6.2.1.7. Endlines
      • 6.2.1.8. Downloads
  • 6.3. File management
    • 6.3.1. Sorting
      • 6.3.1.1. sort-bed
        • 6.3.1.1.1. Inputs and outputs
          • 6.3.1.1.1.1. Input
          • 6.3.1.1.1.2. Output
        • 6.3.1.1.2. Usage
    • 6.3.2. Compression
      • 6.3.2.1. starch
        • 6.3.2.1.1. Inputs and outputs
          • 6.3.2.1.1.1. Input
          • 6.3.2.1.1.2. Output
        • 6.3.2.1.2. Requirements
        • 6.3.2.1.3. Usage
        • 6.3.2.1.4. Options
          • 6.3.2.1.4.1. Backend compression type
          • 6.3.2.1.4.2. Note
          • 6.3.2.1.4.3. Headers
          • 6.3.2.1.4.4. Unique tag
        • 6.3.2.1.5. Example
      • 6.3.2.2. unstarch
        • 6.3.2.2.1. Inputs and outputs
          • 6.3.2.2.1.1. Input
          • 6.3.2.2.1.2. Output
        • 6.3.2.2.2. Requirements
        • 6.3.2.2.3. Usage
          • 6.3.2.2.3.1. Extraction
          • 6.3.2.2.3.2. Archive attributes
            • 6.3.2.2.3.2.1. Metadata
            • 6.3.2.2.3.2.2. Note
            • 6.3.2.2.3.2.3. Timestamp
            • 6.3.2.2.3.2.4. Compression type
            • 6.3.2.2.3.2.5. Version
          • 6.3.2.2.3.3. Whole-file or per-chromosome attributes
            • 6.3.2.2.3.3.1. Elements
            • 6.3.2.2.3.3.2. Bases
        • 6.3.2.2.4. Example
      • 6.3.2.3. starchcat
        • 6.3.2.3.1. Parallelization
        • 6.3.2.3.2. Inputs and outputs
          • 6.3.2.3.2.1. Input
          • 6.3.2.3.2.2. Output
        • 6.3.2.3.3. Usage
          • 6.3.2.3.3.1. Example
      • 6.3.2.4. Starch (v2) specification
        • 6.3.2.4.1. Archive structure
        • 6.3.2.4.2. Magic bytes
        • 6.3.2.4.3. Chromosome streams
        • 6.3.2.4.4. Metadata
          • 6.3.2.4.4.1. Data
            • 6.3.2.4.4.1.1. Archive
            • 6.3.2.4.4.1.2. Streams
          • 6.3.2.4.4.2. Offset
          • 6.3.2.4.4.3. Hash
        • 6.3.2.4.5. Padding
    • 6.3.3. Data conversion
      • 6.3.3.1. bam2bed
        • 6.3.3.1.1. Dependencies
        • 6.3.3.1.2. Source
        • 6.3.3.1.3. Usage
        • 6.3.3.1.4. Example
        • 6.3.3.1.5. Downloads
      • 6.3.3.2. gff2bed
        • 6.3.3.2.1. Dependencies
        • 6.3.3.2.2. Source
        • 6.3.3.2.3. Usage
        • 6.3.3.2.4. Example
        • 6.3.3.2.5. Downloads
      • 6.3.3.3. gtf2bed
        • 6.3.3.3.1. Dependencies
        • 6.3.3.3.2. Source
        • 6.3.3.3.3. Usage
        • 6.3.3.3.4. Example
        • 6.3.3.3.5. Downloads
      • 6.3.3.4. psl2bed
        • 6.3.3.4.1. Dependencies
        • 6.3.3.4.2. Source
        • 6.3.3.4.3. Usage
        • 6.3.3.4.4. Example
        • 6.3.3.4.5. Downloads
      • 6.3.3.5. sam2bed
        • 6.3.3.5.1. Dependencies
        • 6.3.3.5.2. Source
        • 6.3.3.5.3. Usage
        • 6.3.3.5.4. Example
        • 6.3.3.5.5. Downloads
      • 6.3.3.6. vcf2bed
        • 6.3.3.6.1. Dependencies
        • 6.3.3.6.2. Source
        • 6.3.3.6.3. Usage
        • 6.3.3.6.4. Example
        • 6.3.3.6.5. Downloads
      • 6.3.3.7. wig2bed
        • 6.3.3.7.1. Source
        • 6.3.3.7.2. Usage
        • 6.3.3.7.3. Example
        • 6.3.3.7.4. Downloads
• 7. Summary
  • 7.1. Set operation and statistical utilities
    • 7.1.1. bedextract
    • 7.1.2. bedmap
    • 7.1.3. bedops
    • 7.1.4. closest-features
  • 7.2. Sorting
    • 7.2.1. sort-bed
  • 7.3. Compression and extraction
    • 7.3.1. starch
    • 7.3.2. unstarch
    • 7.3.3. starchcat