Makara Documentation

repository·master·Indexed 21 days ago

https://github.com/instacart/makara

A generic primary/replica proxy for Ruby applications that manages connection selection, blacklisting, and cycling. It includes a specialized adapter for ActiveRecord to distribute database load between a primary node and replicas by inspecting SQL strings. Makara supports database-agnostic routing, stickiness to maintain consistency, and specific adapters for MySQL2, PostgreSQL, and PostGIS.

Tokens
8.4K
Snippets
27
Records
40
Agent score
71%

What's inside Makara

  1. Use the ActiveRecord Database Adapter

    master

    Makara acts as a proxy for ActiveRecord, allowing you to distribute database load between a primary node and one or more replicas. It is database-agnostic; it works with any underlying connection (e.g., MySQL, PostgreSQL) by inspecting the SQL string being executed.

    Query Routing Logic

    • SELECT statements: Executed against replica(s).
    • Other statements (Writes/Updates): Executed against the primary.

    Edge Cases and Overrides

    Certain operations bypass replica routing and are sent to the primary:

    • Transactions: All calls inside a transaction are sent to the primary to ensure read-your-writes consistency.
    • Locking reads: Queries like SELECT ... FOR UPDATE are sent to the primary.
    • SET operations: Sent to all connections.
    • Connection management: Methods like connect!, disconnect!, and clear_cache! are invoked on all underlying connections.
  2. How Makara stickiness works

    master

    Stickiness ensures that once a query is sent to the primary, subsequent queries in the same request (or subsequent requests) are also routed to the primary to maintain consistency.

    Configuration Modes

    • sticky: true: Once a query hits the primary, all queries for the remainder of the request go to the primary. Makara sets a client-side cookie (_mkra_stck) with an expiration based on primary_ttl. As long as the cookie is valid, all requests will use the primary.
    • sticky: false: Only queries that explicitly require the primary (via needs_primary?) will go there. Subsequent read queries in the same request will continue to go to replicas.

    Managing Stickiness

    • Forcing Primary: Use proxy.stick_to_primary!(persist) to force the proxy to stick to the primary. If persist is true (default), the stickiness is saved to the context for subsequent requests.
    • Skipping Stickiness: To perform an operation that would normally trigger stickiness without actually sticking the context, wrap the code in a proxy.without_sticking block.
    • Clearing Context: You can release stuck connections using Makara::Context.release_all or for a specific proxy using Makara::Context.release(proxy_id) or Makara::Context.release('name').
    # Force primary and persist across requests
    proxy.stick_to_primary!(true)
    
    # Run code without triggering stickiness
    proxy.without_sticking do
      # logic here
    end
    
    # Clear all stuck connections in the current thread
    Makara::Context.release_all
  3. Handle node blacklisting and errors

    master

    Makara manages node availability through a blacklisting mechanism:

    1. Failure: When a node fails an operation due to a connection issue, it is blacklisted for the duration specified by blacklist_duration.
    2. Recovery: After the duration expires, the node begins receiving queries again.
    3. Fallback: If all replica nodes are blacklisted, the primary node will receive read queries.
    4. Total Failure: If all nodes (including the primary) are blacklisted, the error is raised to the application, and all nodes are simultaneously whitelisted.
  4. Explore the Makara module structure

    master

    Makara is organized into several functional modules. When using the library, you will interact with these primary namespaces:

    • Makara::Proxy: The core component managing connection routing.
    • Makara::Strategies: Contains connection selection algorithms like RoundRobin, PriorityFailover, and ShardAware.
    • Makara::ConfigParser: Handles the parsing of database configurations.
    • Makara::ConnectionWrapper: Wraps individual connections.
    • Makara::Errors: Contains specific error classes for handling connection failures and blacklisting.
    • Makara::Logging: Provides logging and instrumentation via Makara::Logging::Logger and Makara::Logging::Subscriber.
    • Makara::Middleware: Provides middleware components (typically for Rack/Rails integration).
    • Makara::Cache: Handles caching mechanisms used by the proxy.
  5. Manage stickiness state with Makara::Context

    master

    The Makara::Context class manages the stickiness state for different Makara proxies within a single request/thread. It tracks which proxies are currently 'stuck' to the primary database and allows staging changes that will be committed to the persistent stored_data during the next commit or next call.

    Stickiness is managed via proxy_id and a Time-To-Live (ttl).

  6. How Makara::ConnectionWrapper manages connection state

    master

    The Makara::ConnectionWrapper is an internal abstraction that wraps an underlying database connection to manage its lifecycle, metadata, and health within a Makara::Proxy.

    Key responsibilities include:

    • Blacklisting: Automatically blacklisting a node if a connection fails, based on the @config[:blacklist_duration] setting.
    • Metadata Tracking: Storing node-specific information like :name, :weight, and :shard_id from the configuration.
    • Connection Hijacking: Decorating the underlying connection with _makara* methods to allow the Makara::Proxy to intercept specific method calls (hijack methods) or control methods (like those used for ActiveRecord connection pool management).
    • Query Sanitization: Automatically replacing specific SQL strings (e.g., SET client_min_messages TO '') with compatible versions to prevent errors during connection switches.
  7. Use the ShardAware strategy for sharded environments

    master

    The Makara::Strategies::ShardAware strategy is used to manage database connections in a sharded environment. It selects a connection based on a shard_id stored in the current thread.

    Key behaviors:

    • Shard Selection: It retrieves the shard ID from Thread.current['makara_shard_id']. If no ID is present, it falls back to the pool.default_shard.
    • Lifecycle: When a connection is added to the wrapper, the strategy identifies the correct sub-strategy for that specific shard ID and notifies it.
    • Error Handling: If you attempt to call .current or .next on a shard ID that has not been initialized or is invalid, it raises Makara::Errors::InvalidShard.
    # The shard ID is determined by the value in the thread context
    Thread.current['makara_shard_id'] = 'my_shard_id'
    
    # Subsequent Makara calls will use the strategy associated with 'my_shard_id'
  8. Configure custom error matchers for connection handling

    master

    The Makara::ErrorHandler (used by the adapter) can be extended with custom error matchers to identify specific database errors that should be handled "gracefully" (e.g., triggering a reconnection or blacklisting a connection) rather than being treated as "harsh" application errors.

    Custom matchers can be provided as strings that look like regexes. If a matcher is a string in the format /pattern/flags (e.g., /connection refused/i), it will be converted into a Ruby Regexp with the specified flags (e.g., i for ignore case, m for multiline, x for extended).

  9. Implement a custom connection selection strategy

    master

    To implement a custom connection selection strategy in Makara, inherit from Makara::Strategies::Abstract. Your implementation must provide a current method to return the connection that should be used for the current request (the 'sticky' connection) and a next method to determine which connection should be used for the subsequent request (the 'rotation' logic).

    Key lifecycle methods:

    • initialize(pool): Sets the connection pool.
    • init: An explicit constructor hook that can be overridden to perform setup logic.
    • connection_added(wrapper): An optional hook called when a new connection is added to the pool.
    • current: Required. Returns the connection to be used now.
    • next: Required. Returns the connection to be used next.
    module Makara
      module Strategies
        class MyCustomStrategy < Abstract
          def current
            # Return the current sticky connection
          end
    
          def next
            # Return the next connection in rotation
          end
        end
      end
    end
  10. How Makara determines when to use the Primary vs Replica

    master

    The MakaraAbstractAdapter uses SQL pattern matching to decide whether a query should be routed to the primary database or can be safely executed on a replica.

    • Primary (Master) Queries: Queries that involve locking or specific sequence operations are routed to the primary. This includes SELECT ... FOR UPDATE, SELECT ... LOCK IN SHARE MODE, and calls to sequence functions like nextval, currval, lastval, get_lock, etc.
    • Replica (Slave) Queries: Standard SELECT statements are generally routed to replicas.
    • All-Connection Queries: Queries matching SET patterns (e.g., SET ...) are executed on all connections in the pool.
    • Skip Stickiness: Certain metadata queries (e.g., SHOW FIELDS, DESCRIBE, EXPLAIN, PRAGMA) skip stickiness logic.

    Note: sql_master_matchers and sql_slave_matchers are deprecated. Use sql_primary_matchers and sql_replica_matchers instead.

  11. Configure Makara in database.yml

    master

    To use Makara, set your adapter to #{db_type}_makara (e.g., mysql2_makara or postgresql_makara) and provide a makara configuration block.

    Configuration Options

    Top-level makara options:

    • id: An identifier for the proxy. Highly recommended if sticky: true is used, to prevent stickiness from breaking when the configuration changes.
    • blacklist_duration: Seconds a node is blacklisted after a connection failure (default: 5).
    • disable_blacklist: If true, nodes are not blacklisted on error.
    • sticky: If true, a node is stuck to once used during a specific context.
    • primary_ttl: How long the primary context is persisted (should be longer than replication lag).
    • primary_strategy: Strategy for picking the primary node (round_robin [default] or failover).
    • replica_strategy: Strategy for picking the replica node (round_robin [default] or failover).
    • connection_error_matchers: An array of regexes or strings used to identify errors that should trigger blacklisting instead of being raised.

    Connection options:

    Each connection in the connections list can override top-level settings.

    • role: Must be set to primary for the primary node. If omitted, the node is assumed to be a replica.
    • weight: Used to balance traffic (e.g., a node with weight 8 receives more traffic than weight 2).
    • name: An optional attribute used for SQL logging.
    • url: A full connection string (e.g., mysql2://user:pass@host:port/db).

    Warning: Do NOT use the standard ENV['DATABASE_URL'] as it interferes with Makara's initialization. Use custom environment variables instead.

    production:
      adapter: 'mysql2_makara'
      database: 'MyAppProduction'
      makara:
        id: mysql
        blacklist_duration: 5
        primary_ttl: 5
        primary_strategy: round_robin
        sticky: true
        connections:
          - role: primary
            host: primary.sql.host
          - role: replica
            host: replica1.sql.host
            weight: 8
            name: Big Replica
          - role: replica
            host: replica2.sql.host
            weight: 2
            name: Small Replica