kelindar/column

For frequently used queries, you can create an index using CreateIndex(). An index is defined by a column name and a predicate function. Once created, you can query the index using With(indexName), which is significantly faster (often 10-100x) because it uses bitmap indexing and logical operations instead of full scans.

Index Operations:

• Union(otherIndex): Performs a logical OR (merges results).
• Without(otherIndex): Performs a logical AND NOT (difference).
• With(index): Selects rows matching the index.

// Create the index "rogue" in advance
out.CreateIndex("rogue", "class", func(v interface{}) bool {
	return v == "rogue"
})

// This returns the same result as the query before, but much faster
players.Query(func(txn *column.Txn) error {
	count := txn.With("rogue").Count()
	return nil
})

All batch queries must run within a transaction via the Query method. The transaction lifecycle is tied to the return value of the provided callback function:

• Commit: If the function returns nil, the transaction is automatically committed and changes are applied.
• Rollback: If the function returns an error, the transaction is automatically rolled back and no changes are applied.

// Successful commit
players.Query(func(txn *column.Txn) error {
	balance := txn.Float64("balance")
	txn.Range(func(i uint32) {
		balance.Set(10.0)
	})
	return nil // Commit happens here
})

// Automatic rollback on error
players.Query(func(txn *column.Txn) error {
	balance := txn.Float64("balance")
	txn.Range(func(i uint32) {
		balance.Set(10.0)
	})
	return fmt.Errorf("bug") // Rollback happens here
})

To iterate over the rows returned by a query, use the Range() method within a transaction.

Crucial Step: Before calling Range(), you must load column readers for the data you intend to access using methods like txn.String(columnName), txn.Int64(columnName), etc. These readers prepare the necessary buffers for efficient lookup during iteration.

players.Query(func(txn *column.Txn) error {
	names := txn.String("name") // Create a column reader

	return txn.With("rogue").Range(func(i uint32) {
		name, _ := names.Get()
		println("rogue name", name)
	})
})

You can use a custom merging function to handle concurrent updates to the same column atomically. When a transaction is committed, the specified merging function is used to reconcile the existing value with the new update.

This pattern is useful for complex data types, such as JSON-encoded objects containing multiple fields (e.g., position and velocity), where you want to update specific parts of the object without overwriting the entire state. By deferring the merge to the transaction commit phase, the system allows multiple concurrent transactions to prepare updates for the same key, ensuring consistency through the merge logic.

The library automatically manages an expire column for each collection to handle automatic row deletion.

• Setting TTL: Use r.SetTTL(duration) during an Insert to define how long a row should live.
• Extending TTL: Since expire is a standard column, you can use txn.TTL() to get a TTL accessor and call ttl.Extend(duration) within a transaction to add more time to existing rows.

// Set TTL on insert
players.Insert(func(r column.Row) error {
	r.SetString("name", "Merlin")
	r.SetString("class", "mage")
	r.SetTTL(5 * time.Second) // time-to-live of 5 seconds
	return nil
})

// Extend TTL in a query
players.Query(func(txn *column.Txn) error {
	ttl := txn.TTL()
	return txn.Range(func(i uint32) error {
		ttl.Extend(1 * time.Hour) // Add some time
		return nil
	})
})

To store data, you must first create a Collection using NewCollection(). You then define its schema by specifying columns with types like ForString(), ForFloat64(), or ForInt16(). Alternatively, you can use CreateColumnsOf() to automatically infer the schema from an object.

Example of manual schema definition:

// Create a new collection with some columns
players := column.NewCollection()
players.CreateColumn("name", column.ForString())
players.CreateColumn("class", column.ForString())
players.CreateColumn("balance", column.ForFloat64())
players.CreateColumn("age", column.ForInt16())

Collections can be persisted to a binary format using Snapshot() and Restore() methods.

• Snapshot: Call players.Snapshot(io.Writer) to save the current state. This can be done while transactions are running.
• Restore: Call players.Restore(io.Reader) to load a snapshot.

Important: The collection and its schema must be initialized before calling Restore(), as snapshots do not contain schema information.

The standard workflow for using the column package involves four main steps:

1. Create a collection: Initialize a new data collection.
2. Load data: Populate the collection with initial data.
3. Insert indexes: Define indexes on specific columns to enable efficient querying.
4. Query and Iterate: Execute a query against the collection and iterate over the resulting set.

This pattern allows for efficient columnar storage with bitmap indexing for fast lookups.