DOCSEARCHbyAlgolia
Submit Docs

uap-python Documentation

repository·master·Indexed 20 days ago

https://github.com/ua-parser/uap-python

The official Python implementation of the User Agent String Parser project for extracting structured browser, OS, and device information from raw user-agent strings. Version 1.0.2 features a modern API using typed dataclasses, multiple resolver options (including high-performance re2 and Rust-based ua-parser-rs), and various caching algorithms such as S3-FIFO, SIEVE, and LRU. Includes tools for evaluating cache hit rates and resolver performance via the hitrates and bench scripts.

Tokens
14.2K
Snippets
50
Records
73
Agent score
69%

What's inside uap-python

  1. Understand the core data types returned by parsing

    master

    When a user agent is successfully resolved, the library returns objects that are implemented as Python dataclasses. This means you can use standard dataclasses utility functions on them.

    Key data types include:

    • UserAgent: The primary object containing full UA details.
    • OS: Information about the operating system.
    • Device: Information about the device.
    • Result: A successful resolution result.
    • PartialResult: A result that only partially matched the input.
    • DefaultedResult: A result where missing fields are filled with defaults.

  2. Use Eager vs Lazy Matchers

    master

    Matchers can be either Eager or Lazy:

    • Eager Matchers (ua_parser.matchers): Patterns are compiled to re.Pattern objects immediately upon loading. This is most efficient for resolvers that need to apply almost every pattern (like ua_parser.basic.Resolver).
    • Lazy Matchers (ua_parser.lazy): Patterns are compiled only when they are first used. This saves upfront CPU during loading and is highly beneficial when using resolvers that prune matchers (like ua_parser.re2.Resolver), as many patterns may never need to be compiled.

  3. Implement a custom Resolver

    master

    A Resolver is a typing.Protocol (a callable) that takes a user agent string and a Domain, and returns a PartialResult.

    When implementing a resolver:

    • If a domain is requested, you must return it, even if it is None (to signal a match failure).
    • If your resolver finds a match, return a PartialResult containing the matched data.
    • If it doesn't match, return a PartialResult where the requested components are None and the original ua string is preserved.
    from ua_parser import PartialResult, UserAgent, Device, Domain

def my_custom_resolver(ua: str, domain: Domain) -> PartialResult:
    if ua.startswith('my-app/'):
        # Logic for custom parsing
        return PartialResult(domain, UserAgent(family='my-app', major=1), None, None, ua)
    
    # Match failure: must return the domain and None for components
    return PartialResult(domain, None, None, None, ua)

  4. Understand the role of ua-parser-rs

    master

    The ua-parser-rs package is a native accelerator for the main ua-parser Python package. It provides a high-performance Rust-based implementation of the user agent parsing logic.

    Important Note: This package is currently not intended to be used directly by end-users. Instead, it is designed to be used as a backend accelerator for the standard ua-parser library to improve performance.

  5. Optimize parsing with Caching

    master

    If you are using a slower resolver like ua_parser.basic.Resolver and processing many repetitive user agent strings (common in web clients), you should use a CachingResolver to avoid redundant parses.

    ua_parser.caching provides several cache implementations:

    • Lru: Least Recently Used cache.
    • Sieve: A sieve-based cache.
    • Local: Local storage cache.
    • S3Fifo: FIFO cache backed by S3.

    You can wrap a resolver with a cache to improve performance at the cost of memory.

  6. Understand the purpose of ua-parser-builtins

    master
    The ua-parser-builtins package does not provide a standalone API or functional logic. Instead, it serves as a data provider containing the precompiled dataset from uap-core. Its primary purpose is to be used by the ua-parser package to significantly decrease initialization times by providing ready-to-use rulesets.

  7. Compare available cache algorithms

    master

    UA-Parser provides several cache algorithms, each with different trade-offs regarding hit rates, memory usage, and eviction complexity:

    S3-FIFO

    A novel FIFO-based algorithm.

    • Pros: Excellent hit rates, thread-safe on hits, handles "one-hit wonders" and "rare-fews" well.
    • Cons: O(n) eviction, higher memory demand at small sizes.
    • Space Complexity: Consists of one dict (size 1.9n) and three deques (sizes 0.1n, 0.9n, and 0.9n).

    SIEVE

    A FIFO-based algorithm related to S3-FIFO.

    • Pros: Good hit rates, thread-safe on hits, memory efficient.
    • Cons: O(n) eviction, requires linked lists to remove entries from the middle.
    • Space Complexity: Consists of a dict (size n) and a linked list with n nodes (4 pointers each).

    LRU (Least Recently Used)

    The standard non-trivial eviction algorithm.

    • Pros: O(1) eviction, widely understood, available via Python's collections.OrderedDict.
    • Cons: Must be synchronized on hits (entries are moved), generally lower hit rates than FIFO-based novel algorithms.
    • Space Complexity: Consists of an OrderedDict of size n.

  9. Load custom rulesets

    master

    You can use custom rulesets to trim down the default ruleset for efficiency, add rules specific to your own traffic, or use experimental rules. Use ua_parser.loaders to convert external formats to internal data and Parser.from_matchers to create a parser from that data.

    from ua_parser import Parser
from ua_parser.loaders import load_yaml # requires PyYaml

parser = Parser.from_matchers(load_yaml("regexes.yaml"))
parser.parse(some_ua)

  10. Evaluate cache fitness using the hitrates script

    master

    The hitrates command-line script measures the hit rates and memory overhead of ua-parser's available cache implementations. It simulates cache use at various sizes using a provided sample file.

    Requirements: You must provide a sample file that is a representative, unsorted, and undeduplicated sample of your real-world traffic.

    Key Features:

    • Measures hit rates for different cache sizes.
    • Reports memory overhead (total and per entry). Note that overhead does not include the size of the cached entries themselves (typically 500~700 bytes per entry).
    • Includes Bélády's MIN (OPT) algorithm as a theoretical upper bound for reference.
    • Highly efficient as it focuses on cache mechanics rather than full data processing.
    python -mua_parser hitrates <path_to_sample_file>

  11. Extract only browser data using parse_user_agent

    master

    Use parse_user_agent() to extract only the browser-related information. If the user-agent string is empty or cannot be matched, it returns None.

    from ua_parser import parse_user_agent
ua_string = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36'
user_agent = parse_user_agent(ua_string)
# Returns UserAgent(family='Chrome', major='41', minor='0', patch='2272', patch_minor='104')

  12. Retrieve all data from a user-agent string

    master

    Use the parse() function to extract comprehensive information from a user-agent string. It returns a Result object containing UserAgent, OS, and Device objects. If a specific datum cannot be found, its value is set to None.

    from ua_parser import parse
ua_string = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36'
result = parse(ua_string)
# Returns Result(user_agent=UserAgent(...), os=OS(...), device=Device(...), string='...')