ip2region

repository·master·Indexed 12 days ago

https://github.com/lionsoul2014/ip2region

An offline IP address localization library and data management framework supporting IPv4 and IPv6. It utilizes a custom .xdb file format to achieve microsecond-level query performance. The library provides bindings for C, C++, Cangjie, and Golang, offering multiple cache policies including FileOnly, VectorIndex, and ContentBuff to balance memory usage and lookup speed.

Tokens
50.2K
Snippets
189
Records
229
Agent score
97%

What's inside ip2region

  1. Overview of ip2region

    master
    ip2region is an offline IP address localization library and data management framework. It supports both IPv4 and IPv6 with high query efficiency (approximately 10-microsecond level). It uses a custom xdb file format for storing IP segments and provides query client implementations for many mainstream programming languages.
  2. Manage location information with xdb_region_buffer_t

    master

    The xdb_region_buffer_t manages the memory used to store the resulting location string. There are two ways to initialize it:

    1. Fixed Buffer (Pre-allocated): Provide a pre-allocated char array and its size. This is recommended if you know the maximum length of the location string, as it reduces memory fragmentation.
    2. Automatic Allocation: Pass NULL and 0 for size. The library will manage memory allocation automatically. This is useful when the length is uncertain, but it may increase memory fragmentation over long-term operation.

    Critical: You must call xdb_region_buffer_free(&region) after every search to prevent memory leaks.

    // 1. Fixed buffer approach
    char buffer[512];
    xdb_region_buffer_t region;
    int err = xdb_region_buffer_init(&region, buffer, sizeof(buffer));
    
    // 2. Automatic allocation approach
    xdb_region_buffer_t region;
    int err = xdb_region_buffer_init(&region, NULL, 0);
    
    // MUST call this after every search
    xdb_region_buffer_free(&region);
  3. Understand the xdb data format and features

    master

    The xdb format is designed for high-speed IP localization and data management. Key features include:

    • Offline Localization: Provides precise city-level queries. The default field format is Country|Province|City|ISP|iso-alpha2-code. Chinese regions are in Chinese, while non-Chinese regions are in English.
    • Data Management: Supports hundreds of millions of IP segments. The region information is fully customizable; you can append business-specific data like GPS coordinates, postal codes, or international region codes to the standard format.
    • Compression: The xdb generator automatically merges adjacent IP segments and deduplicates/compresses identical region information.
    • High-Speed Querying: Achieves 10-microsecond response times. You can further optimize performance using:
      1. vIndex Index Cache: Uses a fixed 512KiB memory space to cache vector index data, reducing disk IO and keeping average query efficiency under 100 microseconds.
      2. Full File Cache: Loading the entire xdb file into memory (memory usage equals file size) eliminates disk IO for 10-microsecond performance.
  4. Configure IPv4 and IPv6 queries

    master

    To query IP addresses, you must match the version to the specific .xdb file being used. Using a version that does not match the db_path file will cause an error during execution.

    • For IPv4: Use an IPv4 .xdb file and set the version to util.IPv4.
    • For IPv6: Use an IPv6 .xdb file and set the version to util.IPv6.
    import ip2region.util as util
    
    # IPv4 configuration
    db_path = "../../data/ip2region_v4.xdb"
    version = util.IPv4
    
    # IPv6 configuration
    db_path = "../../data/ip2region_v6.xdb"
    version = util.IPv6
  5. Perform memory-based queries by caching the entire xdb file

    master

    For maximum performance, you can load the entire .xdb file into memory. This allows for completely memory-based queries.

    Workflow:

    1. Load content using xdb.load_content(db_path).
    2. Create a searcher using xdb.new_with_buffer(version, content).

    Concurrency Note: Unlike the file-based approach, query objects created with the entire xdb cache can be safely used concurrently. It is recommended to create a single global searcher object at startup and use it globally.

    local xdb = require("xdb_searcher")
    
    -- Load entire file into memory once
    local content = xdb.load_content(db_path)
    
    -- Create a global searcher for concurrent use
    local searcher, err = xdb.new_with_buffer(xdb.IPv4, content)
    
    -- ... query ...
    searcher:close()
    xdb.cleanup()
  6. Perform fully memory-based queries by caching the entire xdb file

    master

    To achieve maximum performance, you can load the entire .xdb file into memory using LongByteArray. This allows for a fully memory-based query. Unlike the VectorIndex method, a Searcher created with the entire buffer can be safely used as a global object for cross-thread access (concurrency-safe).

    // 1. Load entire content into memory
    LongByteArray cBuff = Searcher.loadContentFromFile(dbPath);
    
    // 2. Create a fully memory-based query object
    Searcher searcher = Searcher.newWithBuffer(Version.IPv4, cBuff);
    
    // 3. Query (this searcher can be shared globally across threads)
    String region = searcher.search("1.2.3.4");
    
    // Note: Do not close if shared globally; close when the service shuts down
  7. Cache VectorIndex to reduce IO

    master

    If your environment supports it, you can load the VectorIndex into a global variable. This reduces fixed IO operations and accelerates queries. In concurrent environments, each thread or coroutine should create its own Searcher object, but they can all share the same read-only global vIndex.

    use \ip2region\xdb\Util;
    use \ip2region\xdb\Searcher;
    
    // 1. Load VectorIndex cache
    $vIndex = Util::loadVectorIndexFromFile($dbFile);
    
    // 2. Create searcher with cache
    $searcher = Searcher::newWithVectorIndex($version, $dbFile, $vIndex);
    
    // 3. Query
    $region = $searcher->search('1.2.3.4');
    $searcher->close();
  8. Manage region information with xdb_region_buffer_t

    master

    The xdb_region_buffer_t object manages the memory used to store the resulting region information. There are two ways to initialize it:

    1. Fixed Buffer (Recommended for performance): Provide a pre-allocated buffer (e.g., a char array). This is ideal if you know the maximum length of the region string, as it reduces memory fragmentation.
    2. Automatic Allocation: Pass NULL to xdb_region_buffer_init. The object will automatically manage memory allocation for arbitrary string lengths. Note that this may increase memory fragmentation over long periods of operation.

    Critical: You must call xdb_region_buffer_free(&region) after every search to release the memory. Failure to do so will cause subsequent searches to error out.

    // 1. Using a fixed buffer
    char buffer[512];
    xdb_region_buffer_t region;
    int err = xdb_region_buffer_init(&region, buffer, sizeof(buffer));
    
    // 2. Using automatic allocation
    xdb_region_buffer_t region;
    int err = xdb_region_buffer_init(&region, NULL, 0);
    
    // ALWAYS free after use
    xdb_region_buffer_free(&region);
  9. Configure IPv4 and IPv6 for xdb_searcher

    master

    When initializing a searcher, you must ensure the dbPath matches the IP version specified by the version constant. Mismatched versions will cause errors during query execution.

    • IPv4: Use an IPv4 .xdb file and xdb.IPv4.
    • IPv6: Use an IPv6 .xdb file and xdb.IPv6.
    local xdb = require("xdb_searcher")
    
    -- IPv4 Setup
    local dbPathV4  = "../../data/ip2region_v4.xdb"
    local versionV4 = xdb.IPv4
    
    -- IPv6 Setup
    local dbPathV6  = "../../data/ip2region_v6.xdb"
    local versionV6 = xdb.IPv6
  10. Perform fully memory-based queries

    master

    For maximum performance, you can load the entire .xdb file content into a memory buffer. This enables queries that are entirely memory-resident, similar to the legacy memory search mode.

    Workflow:

    1. Load the file content using Util::loadContentFromFile($dbFile) into a global buffer.
    2. Create a Searcher using Searcher::newWithBuffer($version, $cBuff).

    Concurrency Note: Searcher objects created with a buffer are safe for concurrent use. You can keep the Searcher open for the duration of the service life.

    // 1. Load entire xdb into memory
    $cBuff = Util::loadContentFromFile($dbFile);
    
    // 2. Create searcher using the buffer
    try {
        $searcher = Searcher::newWithBuffer($version, $cBuff);
        $region = $searcher->search('1.2.3.4');
        // $searcher->close(); // Safe to keep open for concurrent use
    } catch (Exception $e) {
        // handle error
    }
  11. Optimize performance with full memory caching (BufferCache)

    master

    You can load the entire .xdb file into memory to perform queries entirely from RAM. This is similar to the legacy memory search implementation.

    Pattern: The cBuff (content buffer) can be shared globally. Searchers created from the same buffer are safe for concurrent use.

    // 1. Load entire xdb into memory
    cBuff, err := xdb.LoadContentFromFile(dbPath)
    if err != nil {
        return err
    }
    
    // 2. Create searchers using the global buffer
    searcher, err := xdb.NewWithBuffer(version, cBuff)
    cBuff, err := xdb.LoadContentFromFile(dbPath)
    searcher, err := xdb.NewWithBuffer(version, cBuff)