VSEARCH Documentation

When vsearch reports failed merges, the reasons fall into two categories:

User-adjustable reasons (controlled by flags):

• reads too short (after truncation): Controlled by --fastq_minlen.
• reads too long (after truncation): Controlled by --fastq_maxlen.
• too many N's: Controlled by --fastq_maxns.
• too many differences: Controlled by --fastq_maxdiffs.
• too high percentage of differences: Controlled by --fastq_maxdiffpct.
• overlap too short: Controlled by --fastq_minovlen.
• expected error too high: Controlled by --fastq_maxee.
• merged fragment too short: Controlled by --fastq_minmergelen.
• merged fragment too long: Controlled by --fastq_maxmergelen.
• staggered read pairs: Occurs if not using --fastq_allowmergestagger.

Internal alignment heuristics (not directly controllable):

• too few kmers found on same diagonal: No candidate overlap could be located via k-mer matching.
• multiple potential alignments: The overlap region is ambiguous (e.g., contains tandem repeats).
• alignment score too low, or score drop too high: The alignment was found but failed internal quality/stability thresholds (often due to indels or clustered mismatches).

The Dbindex API has transitioned from a global singleton model to an RAII-based class model. Instead of using global functions like the_index or dbindex_* free functions, users should now manage a Dbindex instance.

Key Changes

• No more globals: The singleton pattern and global extern declarations have been removed.
• RAII Pattern: Use the Dbindex class to manage the lifecycle of the index.
• Session Initialization: Library session and batch entry points (such as search_session_init, search_batch, cluster_session_init, and chimera_detect_init) now require the caller to pass a reference to a Dbindex object.

API Usage Patterns

• Search/Batch: Pass const Dbindex & to initialization functions.
• Clustering: Pass a mutable Dbindex & to cluster_session_init, as the session adds centroids incrementally.
• Accessing Data: Use member functions on the Dbindex instance (e.g., dbindex.wordlength) rather than global getters.

// Example of the new pattern (conceptual)
Dbindex my_index;
// ... prepare index ...

// Passing the index to a search session
search_session_init(..., my_index);

The expected error (EE) is a quality summary metric for FASTQ reads. It represents the sum of per-base error probabilities across the entire sequence length $L$.

$$EE = P_e(1) + P_e(2) + \dots + P_e(L)$$

Where $P_e(i)$ is the probability that base $i$ was incorrectly sequenced. This probability is derived from the Phred quality score $Q$ using the formula:

$$P_e = 10^{(-Q / 10)}$$

Key Characteristics

• Scale: EE is always greater than zero and at most equal to the sequence length (if every base has an error probability of 1.0).
• Mathematical Advantage: Unlike averaging Phred scores (which is mathematically incorrect because scores are logarithmic), EE sums linear error probabilities, providing a meaningful global quality estimate.
• Poisson Interpretation: Because sequencing errors are approximately independent, EE can be used as the $\lambda$ (lambda) parameter of a Poisson distribution to estimate the probability of observing $k$ errors in a read. For example, a read with $EE = 1.0$ has a 36.8% chance of zero errors and a 36.8% chance of exactly one error.

When working with or extending the line-parsing logic in vsearch, be aware of the following behavioral differences between FASTA and FASTQ formats that the line-scanning abstraction must preserve:

FASTA Parsing

• Sequence Copying: FASTA sequences (Loop B) are copied raw during the loop. Filtering (e.g., removing illegal characters) happens in a second pass via fasta_filter_sequence rather than inline.
• Line Numbering: The FASTA sequence loop does not increment lineno during the scan; newlines are accounted for later via Action::count.
• EOF Policy: A FASTA header must be terminated with a newline; otherwise, it is treated as a fatal error.

FASTQ Parsing

• Inline Filtering: FASTQ sequences and quality scores are filtered inline during the copy process using buffer_filter_extend.
• Line Numbering: All FASTQ loops (header, sequence, + line, and quality) increment lineno upon encountering an LF.
• EOF Policy: Unexpected EOF in headers, sequences, or the + line is treated as a fatal error (fastq_fatal).
• Record Boundaries: The quality loop (Loop F) includes an extra guard to break early if the quality string length exceeds the sequence length.

The --cluster_size command performs abundance-based greedy clustering (AGC). It groups FASTA sequences into clusters using a heuristic, centroid-based algorithm.

How it works:

1. Sorting: Input sequences are automatically sorted by decreasing abundance.
2. Centroid Selection: For each query sequence (from most to least abundant), vsearch compares it to existing cluster centroids using global pairwise alignment (Needleman-Wunsch).
3. Assignment: If the query is similar enough to a centroid (defined by --id), it joins that cluster. Otherwise, the query becomes the centroid of a new cluster.
4. Precedence: Because sequences are processed by abundance, more abundant sequences are more likely to become centroids.

Abundance Information: To use abundance-based clustering, abundance information must be present in the FASTA headers (e.g., ;size=integer;). You must use the --sizein option to tell vsearch to read this information. If no abundance information is found, the behavior defaults to --cluster_fast (sorting by length instead of abundance).

Comparison to other clustering modes:

• --cluster_fast: Sorts by decreasing length instead of abundance.
• --cluster_smallmem: Expects the input to be already sorted (skips the initial sorting step).

vsearch --cluster_size sequences.fasta --id 0.97 --sizein --sizeout --centroids centroids.fasta