Bleve Search Library

Bleve is a modern indexing and search library for Go that supports a wide range of data types and query capabilities:

Supported Field Types:

• text, number, datetime, boolean, geopoint, geoshape, IP, vector

Supported Query Types:

• Term, phrase, match, match_phrase, prefix, regexp, wildcard, fuzzy
• Range queries: term, numeric, and date ranges
• Boolean field queries
• Compound queries: conjuncts, disjuncts, and boolean (must/should/must_not)
• Query string syntax
• Geo spatial search
• Approximate k-nearest neighbors via vector search
• Synonym search
• Hierarchical nested search

Scoring and Advanced Features:

• Scoring models: tf-idf and bm25
• Hybrid search: exact + semantic (supports RRF Reciprocal Rank Fusion and RSF Relative Score Fusion)
• Result pagination and query time boosting
• Match highlighting with document fragments
• Aggregations/faceting: terms, numeric range, and date range facets

DocValues are an optional index feature that can be enabled or disabled per field mapping.

When to enable DocValues

You should enable docvalues: true in your field mapping if you anticipate queries that involve:

1. Sorting: Specifically using SortField or SortGeoDistance types.
2. Faceting: Using Date range, Numeric range, or Term facet requests.

Trade-offs

• Pros: Enhanced query response times (lower latency) and reduced memory consumption during active usage. Instead of fetching full documents to extract field values, Bleve can access values directly from the stored docValue part of the index.
• Cons: Increased disk usage. Enabling docValues will always result in an increase in the size of your Bleve index.

When DocValues are NOT used

• Sorting by SortDocID or SortScore (the default).
• Search requests that do not specify a sort field.
• Search requests that do not include a facet object.

Hierarchical nested search allows you to represent and query complex, multi-level data structures (like a company containing departments, which in turn contain employees and projects) within a single document.

Key Behaviors:

• Context-Aware Conjunctions: Queries like AND (conjunction) respect nested boundaries. Terms must exist within the same nested object to match. For example, searching for name: Alice AND role: Engineer will only match if both fields belong to the same employee object.
• Field-Level Highlighting: Highlighting is restricted to the matched nested object, ensuring the context is correct.
• Nested Faceting/Aggregations: Facets are computed within the matched nested objects, providing context-aware buckets (e.g., project status facets scoped to specific departments).
• Sorting: You can sort by fields within a nested object (e.g., sorting by departments.budget).
• Vector Search:
  • Standard KNN: Scoring is identical for nested and non-nested arrays; the highest-scoring vector is selected and its score is bubbled up to the parent.
  • Pre-Filtered Vector Search: When combining vector search with filters on nested fields, the filters are applied to the nested items first. Vector similarity is then only computed for the subset of nested objects that satisfy the filter.

Bleve supports several query types to handle different search requirements:

1. Query String Query: A simple query that parses a string of terms.
2. Match Query: Searches for an exact term within a specific field using query.SetField("fieldName").
3. Boolean Query: A complex query that combines multiple queries using logic like AddMust (AND) and AddShould (OR).
4. Numeric Range Query: Searches for values within a specific range (numeric or date) using bleve.NewNumericRangeQuery(&min, &max) and query.SetField("fieldName").

// 1. Query String Query
query := bleve.NewQueryStringQuery("golang programming")

// 2. Match Query
query := bleve.NewMatchQuery("bleve")
query.SetField("title")

// 3. Boolean Query
mustQuery := bleve.NewMatchQuery("golang")
shouldQuery := bleve.NewMatchQuery("programming")
boolQuery := bleve.NewBooleanQuery()
boolQuery.AddMust(mustQuery)
boolQuery.AddShould(shouldQuery)

// 4. Range Query
minPrice := 20.50
maxPrice := 40.75
query := bleve.NewNumericRangeQuery(&minPrice, &maxPrice)
query.SetField("price")

For large datasets or deep navigation, use SearchAfter (forward) or SearchBefore (backward) pagination. This method keeps resource usage proportional to the page size rather than the depth of the page.

Rules for usage:

• Use either SearchAfter or SearchBefore, but never both in a single request.
• The length of the search_after or search_before array must exactly match the length of the sort array.
• Values in the array must be strings representing the sort keys in the same order as defined in sort.
• You must maintain the same query and sort parameters across all pages to ensure consistent navigation.

How to get sort keys:

• For each hit in the result set, Bleve provides a Sort array.
• For forward pagination (SearchAfter), take the sort keys from the last hit of the current page.
• For backward pagination (SearchBefore), take the sort keys from the first hit of the current page.

Handling non-string types (Numeric, Datetime, Geo): Internal representations of numeric, datetime, or geo data can appear garbled in the standard Sort field. To use these as pagination keys, use the DecodedSort field (available in Bleve v2.5.2+).

When using DecodedSort, your sort array must explicitly declare the field type using SortField (for numeric/datetime) or SortGeoDistance (for geo) objects instead of simple field name strings.

// Forward pagination example
{
  "query": { "match": "California" },
  "sort": ["_id", "_score"],
  "search_after": ["hotel_10180", "0.998"],
  "size": 3
}

// Backward pagination example
{
  "query": { "match": "California" },
  "sort": ["_id", "_score"],
  "search_before": ["hotel_17595", "0.623"],
  "size": 4
}

// Pagination with complex types (Numeric, Date, Geo)
{
  "query": {
    "match_all": {}
  },
  "size": 10,
  "sort": [
    {"by": "field", "field": "price", "type": "number"},
    {"by": "field", "field": "created_at", "type": "date"},
    {"by": "geo_distance", "field": "location", "location": {"lat": 40.7128, "lon": -74.0060}}
  ],
  "search_after": ["99.99", "2023-10-15T10:30:00Z", "5.2"]
}

Bleve supports a new spatial field type called geoshape which allows you to index complex GeoJSON shapes. This type unblocks support for standard GeoJSON types like Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, and GeometryCollection, as well as Bleve-specific shapes like Circle and Envelope.

To specify a geoshape, use a nested field containing:

• type: The GeoJSON object type (case-insensitive).
• coordinates: The object's coordinates.

Important Requirements:

• Coordinate Order: Always list longitude first, then latitude.
• Longitude Range: -180 to 180 (inclusive).
• Latitude Range: -90 to 90 (inclusive).
• Polygons: The first and last coordinates must match to close the polygon, and exterior coordinates must be in Counter Clockwise Order (CCW).
• Antimeridian: It is strongly suggested to split geometries so they do not cross the antimeridian.

"fieldName": { 
     "type": "GeoJSON Type", 
     "coordinates": <coordinates> 
}

Once synonyms are indexed, any text-based Bleve query (match, phrase, term, fuzzy, etc.) performed on a field with an assigned synonym source will automatically expand the search terms using the defined thesaurus.

Behavior for specific query types:

• Fuzzy Queries (match, phrase): Queried terms are fuzzily matched against the thesaurus's Left-Hand Side (LHS) terms to generate candidates, which are then combined with standard fuzzy dictionary matches.
• Wildcard, Regexp, and Prefix Queries: The thesaurus is used to expand terms (e.g., finding LHS terms that match the prefix/regex) before combining them with dictionary expansion results.

// Create a match query for a term that has synonyms
query := bleve.NewMatchQuery("persistent")
query.SetField("text")

// Execute search
searchRequest := bleve.NewSearchRequest(query)
searchResult, err := index.Search(searchRequest)
// If 'persistent' is a synonym for 'hardworking', 'doc1' containing 'hardworking' will be returned.