DOCSEARCHbyAlgolia
Submit Docs

FeatureBase Documentation

repository·master·Indexed 25 days ago

https://github.com/featurebasedb/featurebase

A real-time distributed database built on bitmaps for low-latency queries on fresh data. It supports batch and streaming ingestion, SQL and PQL querying, and includes a Go client for Pilosa, a data generation tool (datagen), and various consumers for GitHub and Kafka SASL/SSL.

Tokens
39.5K
Snippets
62
Records
263
Agent score
82%

What's inside FeatureBase

  1. Key capabilities of FeatureBase

    master

    FeatureBase is a real-time distributed database built on bitmaps. Key features include:

    • Query Languages: Supports both Pilosa Query Language (PQL) and SQL.
    • Stream and Batch Ingest: Ability to combine real-time data streams with batch historical data.
    • Mutability: Supports scaleable inserts, updates, and deletes in real time.
    • Multi-Valued Set Fields: Allows storing multiple comma-delimited values in a single field while maintaining high performance for counts and TopKs.
    • Time Quantums: Enables ranged Row queries down to a specified granularity (e.g., YMD) by creating extra views.
    • RBF Storage Backend: A compressed bitmap format providing ACID support per shard, reduced memory allocation, and concurrent backups/writes. Note: Pilosa backup files are not compatible with the RBF backend.

  2. Handle Pilosa Query Responses

    master

    When calling cli.Query(), the server returns either a pilosa.Error or a QueryResponse.

    A QueryResponse can contain multiple QueryResult objects. You can access them via:

    • Results(): Returns a list of all QueryResult objects.
    • Result(): Returns the first result or nil if no results exist.

    QueryResult provides methods to extract data based on the query type:

    • Row(): Retrieves a row result.
    • CountItems(): Retrieves column count per row ID (for TopN queries).
    • Count(): Retrieves the number of rows per row ID (for Count queries).
    • Value(): Retrieves the result of Min, Max, or Sum queries.
    • Changed(): Returns whether a Set or Clear query modified a column.
    response, err := cli.Query(field.Row(5))
if err != nil {
    // Act on the error
}

// check that there's a result and act on it
result := response.Result()
if result != nil {
    // Act on the result
}

// iterate over all results
for _, result := range response.Results() {
    // Act on the result
}

  3. Understand the Roaring B-tree Format (RBF) file structure

    master

    The RBF format represents a Roaring bitmap where containers are stored in the leaves of a B-tree. This structure enables efficient querying and updating of the bitmap.

    File Layout

    • Page Size: The file is divided into equal 8KB pages.
    • Page Numbering: After the meta page, pages are numbered incrementally from $1$ to $2^{31}$.
    • Endianness: All integer values are little endian encoded.

    Page Types

    • Meta page: Contains global header information.
    • Branch page: Contains pointers to lower branch and leaf pages.
    • Leaf page: Contains array and RLE (Run-Length Encoding) container data.
    • Bitmap page: Contains bitmap container data (occupies the full 8KB).

  4. Define the Data Model using Indexes and Fields

    master

    The data model is composed of Index and Field objects. Note that creating these objects in the client only defines the schema locally; it does not automatically create them on the server.

    To define an index, use schema.Index(). You can pass options like pilosa.OptIndexKeys(true) to configure the index.

    To define a field within an index, use the Field() method on an index instance. You can specify field types using options like pilosa.OptFieldTypeTime() or pilosa.OptFieldTypeInt().

    schema := client.NewSchema()
repository := schema.Index("repository", pilosa.OptIndexKeys(true))

stargazer := repository.Field("stargazer", pilosa.OptFieldTypeTime(TimeQuantumYearMonthDay))

  5. Use the batch package for performant data ingestion

    master

    The batch package provides tools for batching records to optimize ingestion performance into FeatureBase. The primary implementation is the Batch type, which is instantiated using the NewBatch() function.

    To use NewBatch(), you must provide an Importer. The Importer interface must contain the necessary methods for interacting with FeatureBase, specifically those for:

    • String/ID translation
    • Importing shards of data

    This package is used internally by IDK and the sql3 package (where it handles key translation and batch building during INSERT INTO operations).

  6. Quickstart: Using the Pilosa Go Client

    master

    This example demonstrates how to initialize a client, synchronize a schema (creating indexes and fields), and execute single and batch queries against a running Pilosa server (defaulting to localhost:10101).

    Key steps:

    1. Initialize with client.DefaultClient().
    2. Define your schema using cli.Schema().
    3. Synchronize the schema to the server using cli.SyncSchema(schema).
    4. Execute queries using cli.Query() with methods like .Set(), .Row(), or .BatchQuery().
    5. Process results using response.Result() for single queries or response.Results() for batch queries.
    package main

import (
	"fmt"

	"github.com/pilosa/pilosa/v2/client"
)

func main() {
	// Create the default client
	cli := client.DefaultClient()

	// Retrieve the schema
	schema, err := cli.Schema()

	// Create an Index object
	myindex := schema.Index("myindex")

	// Create a Field object
	myfield := myindex.Field("myfield")

	// make sure the index and the field exists on the server
	err := cli.SyncSchema(schema)

	// Send a Set query. If err is non-nil, response will be nil.
	response, err := cli.Query(myfield.Set(5, 42))

	// Send a Row query. If err is non-nil, response will be nil.
	response, err = cli.Query(myfield.Row(5))

	// Get the result
	result := response.Result()
	// Act on the result
	if result != nil {
		columns := result.Row().Columns
		fmt.Println("Got columns: ", columns)
	}

	// You can batch queries to improve throughput
	response, err = cli.Query(myindex.BatchQuery(
		myfield.Row(5),
		myfield.Row(10)))
	if err != nil {
		fmt.Println(err)
	}

	for _, result := range response.Results() {
		// Act on the result
		fmt.Println(result.Row().Columns)
	}
}

  7. Deploy testing environments with Terraform

    master

    This directory provides Terraform configurations to deploy testing environments either ad-hoc or as part of CI/CD pipelines. The core logic is located in the .modules directory, while specific environment configurations are found in the subdirectories (each containing its own README).

    # To spin up a cluster
terraform plan
terraform apply

# To tear down a cluster
terraform destroy

  8. Execute queries using Index and Field methods

    master

    Queries are constructed using methods attached to Index and Field objects.

    • Row queries: Use Field.Row(rowID) to target specific rows.
    • Set operations: Use Index.Union(), Index.Intersect(), Index.Difference(), or Index.Xor() to combine row queries.
    • Raw queries: If you need to send a manual PQL string, use Index.RawQuery(). Note that raw queries are sent only to the coordinator node and are not validated by the client before sending.

  9. Use the Datagen Tool to generate data

    master

    The datagen tool is used to generate synthetic data and send it to either a Pilosa index or a Kafka topic. You can specify the source of the data, the starting and ending IDs, and the destination target.

    To generate data for a Pilosa index, use the --target=pilosa flag (which is the default) and provide the index name via --pilosa.index.

    datagen --source=equipment --pilosa.index=equipment --end-at=99

  11. Retrieve an existing schema from the server

    master

    If the schema already exists on the server, you can retrieve it using the client's Schema() method instead of creating a new one locally. This ensures your client-side objects match the server's state.

    cli := client.DefaultClient()
schema, err := cli.Schema()
if err != nil {
    // act on the error
}
repository := schema.Index("repository")

  12. Configure Lattice to connect to a Pilosa instance

    master

    When running Lattice in standalone mode, you must configure the connection to your Pilosa server in the src/lattice.config.js file.

    By default, Lattice attempts to use the browser URL to find the server. To point to a specific server, update the hostname and port in src/lattice.config.js. The default Pilosa server address is localhost:10101.

    Tip: If you are developing locally and want to avoid accidentally committing your local configuration changes, you can tell Git to ignore changes to this file using: git update-index --assume-unchanged src/lattice.config.js