repository·master·Indexed 20 days ago
https://github.com/python-happybase/happybaseA high-level, Pythonic library for interacting with Apache HBase via the HBase Thrift server. HappyBase provides a developer-friendly interface through primary classes including Connection for server management, Table for data retrieval and manipulation, Batch for efficient bulk operations, and ConnectionPool for thread-safe connection reuse.
thriftpy2 library to connect to HBase through its Thrift gateway (included in standard HBase 0.9x releases), providing a Pythonic interface for HBase operations.The HappyBase API is structured around four primary classes that manage the lifecycle of a connection to the HBase Thrift server and data operations:
happybase.Connection: The main entry point. Use this to connect to the HBase Thrift server and perform table management tasks.happybase.Table: The primary interface for data interaction. You obtain Table instances via Connection.table(). It provides methods for retrieving and manipulating data.happybase.Batch: Used for efficient data manipulation. You obtain Batch instances via Table.batch().happybase.ConnectionPool: A thread-safe implementation that allows applications to reuse multiple connections efficiently.The HBase Thrift API is a flat, language-agnostic interface tied to wire-level RPC protocols. Using it directly in Python requires managing numerous low-level components such as sockets, transports, protocols, clients, and specific Thrift mutation objects. This results in verbose, cumbersome, and error-prone code.
HappyBase acts as a wrapper that hides this 'Thrift cruft' behind a high-level, Pythonic API. It simplifies common tasks like connecting to HBase, accessing tables, and performing mutations (like put), making application code cleaner, more maintainable, and more productive.
HBase treats all row keys, column names, and column values as raw byte strings. HappyBase does not perform automatic string conversion. To avoid asymmetric behavior (where data is encoded on write but not decoded on read), you must explicitly encode your data to bytes in your application before passing it to HappyBase.
Common practice involves using .encode('utf-8') for text strings or struct.pack() for complex serialization.
To develop on HappyBase, clone the repository and set up a virtual environment. You must install the test requirements before installing the package in editable mode.
$ git clone https://github.com/wbolster/happybase.git
$ cd /path/to/happybase/
$ mkvirtualenv happybase
(happybase)$ pip install -r test-requirements.txt
(happybase)$ pip install -e .To confirm that HappyBase was installed correctly, attempt to import the module in a Python shell. If no errors are returned, the installation was successful.
(envname) $ python -c 'import happybase'HappyBase provides a Pythonic API to interact with Apache HBase via the Thrift gateway. You can establish a connection, access tables, perform CRUD operations (put, row, delete), and iterate through rows using rows() or scan() with prefixes.
import happybase
# Establish a connection to the HBase Thrift server
connection = happybase.Connection('hostname')
# Access a specific table
table = connection.table('table-name')
# Put data into a row (keys and values must be bytes)
table.put(b'row-key', {b'family:qual1': b'value1',
b'family:qual2': b'value2'})
# Retrieve a single row
row = table.row(b'row-key')
print(row[b'family:qual1']) # prints b'value1'
# Retrieve multiple specific rows
for key, data in table.rows([b'row-key-1', b'row-key-2']):
print(key, data) # prints row key and data for each row
# Scan rows with a specific prefix
for key, data in table.scan(row_prefix=b'row'):
print(key, data) # prints row key and data for each row
# Delete a row
table.delete(b'row-key')To connect to HBase, create an instance of happybase.Connection by providing the host address.
Key Configuration Arguments:
compat: Set this if you are using HBase 0.90.x to ensure the correct wire protocol.transport: Specify this if you are using HBase 0.94 with a non-standard Thrift transport mode.autoconnect: If set to False, the connection will not open automatically. You must call .open() manually before performing operations.table_prefix: Used to manage 'namespaces'. HappyBase will prepend this prefix (and an underscore) to all table names used by this connection and strip it when returning table names.import happybase
# Standard connection
connection = happybase.Connection('somehost')
# Connection with manual opening
connection = happybase.Connection('somehost', autoconnect=False)
connection.open()
# Connection with a table namespace prefix
connection = happybase.Connection('somehost', table_prefix='myproject')It is recommended to install HappyBase and Thrift within a virtual environment. You can use virtualenv or virtualenvwrapper to create and activate one.
# Using virtualenv
$ virtualenv envname
$ source envname/bin/activate
# Using virtualenvwrapper
$ mkvirtualenv envnameFor multi-threaded environments (like web servers), use happybase.ConnectionPool. It provides a thread-safe way to share and reuse connections.
Best Practices:
with pool.connection() as connection: to ensure they are returned to the pool.with block to a minimum (e.g., only fetching data). Process the data outside the block to release the connection back to the pool as quickly as possible.with block has ended; it may have already been reassigned to another thread.# Setup the pool
pool = happybase.ConnectionPool(size=3, host='...', table_prefix='myproject')
# Correct usage pattern
with pool.connection() as connection:
table = connection.table('table-name')
row = table.row(b'row-key')
# Process data outside the connection block
process_data(row)Using table.batch() as a context manager is the recommended way to handle batches. It automatically calls send() when the block exits.
To ensure the batch is only sent if no exceptions occur (simulating a transaction), use the transaction=True argument.
# Standard context manager (sends even if error occurs)
with table.batch() as b:
b.put(b'row-key-1', {b'cf:col1': b'value1'})
# Transactional context manager (only sends if no error occurs)
try:
with table.batch(transaction=True) as b:
b.put(b'row-key-1', {b'cf:col1': b'value1'})
raise ValueError("Something went wrong!")
except ValueError:
# The batch was NOT sent to HBase
pass