ctypeslib2

repository·master·Indexed 42 days ago

https://github.com/trolldbois/ctypeslib

An FFI toolkit that uses libclang to translate C headers and source files into Python modules using the ctypes library. It provides the clang2py CLI tool for generating Python code from C headers and supports cross-platform, cross-architecture memory representation by explicitly handling alignment and padding.

Tokens
3.4K
Snippets
9
Records
12
Agent score
46%

What's inside ctypeslib

  1. How ctypeslib handles memory alignment and padding

    master

    To support cross-architecture memory reading (e.g., reading a memory dump from a different CPU architecture), ctypeslib uses two mechanisms:

    1. _pack_ = True: The generated ctypes.Structure classes include the _pack_ attribute set to True. This forces alignment on 0 bytes, ensuring that field offsets match the original C definition regardless of the host Python binary's alignment rules.
    2. PADDING_n fields: If the Clang compiler inserts padding to align fields, ctypeslib explicitly adds these as PADDING_n fields in the Python structure. This ensures the ctypes memory layout exactly matches the C memory layout.

    This approach allows a Python script running on one architecture to correctly interpret data structures defined for another.

  2. Understand the purpose of this ctypeslib fork

    master

    This fork of ctypeslib is specifically designed to use libclang1>=3.7 python bindings to generate Python code from C source code. Unlike the original ctypeslib which relied on gccxml, this version is heavily patched to work with Clang.

    It maintains the original package structure:

    • ctypeslib.codegen: The code generator.
    • ctypeslib.contrib: Contributed modules.
    • ctypeslib.util: Helper functions.
    • ctypeslib.test: Unittests.
  3. Install ctypeslib2 and LLVM Clang

    master

    To use ctypeslib2, you must first install the LLVM Clang library on your system, followed by the ctypeslib2 Python package and the clang Python bindings. It is critical that the version of the clang Python package matches the version of the LLVM Clang library installed on your system.

    1. Install LLVM Clang

    Linux (APT):

    $ sudo apt install libclang1-19

    Using LLVM install script:

    $ wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh
    $ ./llvm.sh 19

    macOS: Install Xcode Command Line Tools:

    $ xcode-select --install

    2. Install Python Packages

    Install ctypeslib2 and the corresponding clang package via pip. Ensure the version numbers match your installed LLVM version.

    LLVM VersionPip Command
    Latestpip install ctypeslib2
    Clang 16pip install ctypeslib2 clang==16
    Clang 14pip install ctypeslib2 clang==14
    Clang 11pip install ctypeslib2 clang==11

    macOS specific command (for Clang 15):

    % pip install ctypeslib2 'clang>=15,<16'
    # Example for Clang 16
    pip install ctypeslib2 clang==16
  4. Use the clang2py CLI to generate Python code from C headers

    master

    The clang2py CLI tool generates Python-friendly interfaces to C libraries by converting Clang headers into ctypes definitions. It can automatically generate Python classes representing C data structures and simplify wrapping C functions.

    Basic Usage

    clang2py [options] <source_filenames>

    If no output filename is specified via -o, the generated code is sent to standard output (stdout). You can also provide - as a filename to explicitly use stdout.

    Key CLI Options

    Generation Control

    • -c, --comments: Include source Doxygen-style comments in the output.
    • -d, --doc: Include docstrings containing the C prototype and source file location.
    • -e, --show-definition-location: Include the source file location in comments.
    • -k, --kind <TYPEKIND>: Specify which type descriptions to include.
      • Valid characters: a (Alias), c (Class), d (Variable), e (Enumeration), f (Function), m (Macro), s (Structure), t (Typedef), u (Union).
      • Default: cdefstu.
    • -i, --includes: Include declarations defined outside of the provided source files.
    • -x, --exclude-includes: Parse objects in source files only; ignore includes.

    Symbol and Library Filtering

    • -s, --symbol <SYMBOL>: Include only specific symbols. If no symbols or regexes are provided, everything is included.
    • -r, --regex <EXPRESSION>: Include symbols matching the provided regular expression.
    • -l, --include-library <DLL>: Specify a library to search for exported functions. This can be used multiple times.
    • -w: Add all standard Windows DLLs to the searched DLL list.
    • -m, --module <module>: Specify Python modules containing symbols that should be imported instead of being generated.
    • --preload <DLL>: Specify a DLL to be loaded before all others to resolve symbols.
    • --nm <program>: The nm program to use for extracting symbols from libraries (defaults to nm).

    Configuration and Debugging

    • -o, --output <filename>: Specify the output filename (defaults to - for stdout).
    • --clang-args="<args>": Pass raw arguments directly to Clang (e.g., --clang-args="-std=c99 -Wall").
    • -t, --target <target>: Set the target architecture (e.g., --target x86_64).
    • --debug: Set logging level to DEBUG.
    • -q, --quiet: Shut down warnings and below.
    • -v, --verbose: Enable verbose output.
    • --max-depth <N>: Limit cursor expansion to depth N.
    • --show-ids: Do not compute cursor IDs (improves speed but may limit functionality).
    • --validate <bool>: Validate the generated Python code (defaults to True).

    Input/Output

    • Input Files: Provide one or more source filenames. Use - to read from stdin.
    • Output: Use -o <file> to save to a file, or omit to use stdout.
    # Example: Generate Python code from a header, including docstrings and comments, saving to a file
    clang2py -c -d -o my_library.py my_header.h
    
    # Example: Generate code for specific symbols using a regex and passing clang flags
    clang2py -r "^api_" --clang-args="-I/usr/include/custom" my_header.h
  5. Convert C headers to Python ctypes code

    master

    ctypeslib transforms C headers into ABI-compatible Python code using ctypes. The process involves two main steps: parsing the C headers using clangparser.Clang_Parser and generating Python source code using codegenerator.Generator.

    import ctypeslib
    from ctypeslib.codegen import clangparser
    from ctypeslib.codegen import codegenerator
    from StringIO import StringIO
    
    # 1. Define C headers
    c_headers = '''
    int version = 2;
    char text[] = "Hello World!";
    '''
    
    # 2. Parse the headers
    parser = clangparser.Clang_Parser(flags=[])
    parser.parse_string(c_headers)
    items = parser.get_result()
    
    # 3. Generate Python code
    output = StringIO()
    gen = codegenerator.Generator(output)
    gen.generate(parser, items)
    
    # 4. Access the generated code
    print output.getvalue()
  6. Troubleshoot Clang library loading on Ubuntu/Debian

    master

    If you encounter version compatibility issues on Ubuntu or Debian where the library cannot be found, use one of the following methods:

    1. Set the CLANG_LIBRARY_PATH environment variable to point directly to your .so file:
      export CLANG_LIBRARY_PATH=/lib/x86_64-linux-gnu/libclang-11.so.1
    2. Install the development package to provide a generic libclang.so symlink:
      sudo apt get install libclang-<version>-dev
    3. Manually link the library: Create a symlink from libclang-<version>.so.1 to libclang.so.
    4. Hardcode the path in Python: Call clang.cindex.Config.load_library_file('libclang-<version>.so.1') in your code before importing ctypeslib.
  7. Use ctypeslib2 as a Python library

    master

    You can use ctypeslib directly in your Python code to translate C strings or files into Python modules using ctypes structures.

    Translate C strings

    Use ctypeslib.translate() to convert a C code snippet into a Python object.

    Translate C files

    Use ctypeslib.translate_files() to convert one or more .c files. You can specify an outfile to write the result to a file or use cfg (a CodegenConfig object) to pass custom Clang arguments for cross-platform translation.

    import ctypeslib
    from ctypeslib.codegen import config
    
    # 1. Simple translation
    py_module = ctypeslib.translate('''int i = 12;''')
    print(py_module.i)
    
    # 2. Translate files to a specific output file
    ctypeslib.translate_files(['mytest.c'], outfile=open('mytest.py', 'w'))
    
    # 3. Translate with custom Clang arguments (e.g., for cross-compilation)
    cfg = config.CodegenConfig()
    cfg.clang_opts.extend(['-target', 'arm-gnu-linux'])
    py_module = ctypeslib.translate_files(['mytest.c'], cfg=cfg)
    import ctypeslib
    
    # Translate a string
    py_module = ctypeslib.translate('''
    struct coordinates { int i; int y; };
    ''')
    
    # Access the generated structure
    coords = py_module.struct_coordinates(1, 2)
    print(coords.i, coords.y)
  8. Load generated ctypes code into a Python namespace

    master

    To use the generated code immediately in your script, you can execute the generated string within a dictionary (namespace) and wrap it with ctypeslib.codegen.util.ADict for easier attribute access.

    Note: The generated code includes __all__ and defines structures/variables that map to ctypes types.

    from ctypeslib.codegen import util
    
    # Assuming 'output' is a StringIO containing the generated code
    namespace = {}
    exec output.getvalue() in namespace
    
    # Wrap in ADict for dot-notation access (e.g., namespace.struct_Name)
    namespace = util.ADict(namespace)
    
    # Now you can use the structures
    instance = namespace.struct_Name()
    print(instance)
  9. Use the clang2py CLI tool

    master

    The clang2py command-line tool generates Python code from C header or source files. It is useful for batch processing or quick conversions.

    Basic Usage

    clang2py my_header.h

    Common CLI Flags

    • -o OUTPUT: Specify the output filename (defaults to stdout).
    • --clang-args CLANG_ARGS: Pass raw arguments to Clang (e.g., --clang-args="-I/usr/include/clang/4.0/include").
    • -t TARGET: Specify the target architecture (e.g., -t x86_64-Linux).
    • -s SYMBOL: Include only specific symbols.
    • -r EXPRESSION: Use a regex to include specific symbols.
    • -k TYPEKIND: Filter which type descriptions to include (e.g., s for Structures, f for Functions).

    Cross-Architecture Translation

    To generate code for a different architecture, use the --clang-args flag to pass the -target option to Clang:

    clang2py --clang-args="-target x86_64" my_file.c
    # Example: Generate python code from a C file with specific include paths
    clang2py --clang-args="-I/usr/include/clang/4.0/include" test-stdbool.c
  10. Parse C headers with Clang_Parser

    master

    The clangparser.Clang_Parser class is used to parse C header strings or files. It requires the LLVM Clang library and its Python bindings to be installed on the system.

    • Clang_Parser(flags=[]): Initializes the parser. You can pass Clang flags as a list to influence the parsing behavior.
    • .parse_string(c_headers): Parses a string containing C code.
    • .get_result(): Returns a collection of intermediate Python objects representing the parsed C entities.
    from ctypeslib.codegen import clangparser
    
    parser = clangparser.Clang_Parser(flags=[])
    parser.parse_string("int x = 10;")
    items = parser.get_result()
  11. Generate Python source code with Generator

    master

    The codegenerator.Generator class converts the intermediate objects produced by the parser into executable Python source code.

    • Generator(output_stream): Initializes the generator with an output stream (e.g., StringIO or a file object).
    • .generate(parser, items): Takes the parser instance and the items returned by parser.get_result() to write the generated code into the provided output stream.
    from ctypeslib.codegen import codegenerator
    from StringIO import StringIO
    
    output = StringIO()
    gen = codegenerator.Generator(output)
    gen.generate(parser, items)
  12. Reference: clang2py type description kinds

    master

    When using the --kind (or -k) flag, you can control which C types are translated into Python. The following characters represent the supported types:

    • a: Alias
    • c: Class
    • d: Variable
    • e: Enumeration
    • f: Function
    • m: Macro (#define)
    • s: Structure
    • t: Typedef
    • u: Union

    Default value is cdefstu.

    a = Alias
    c = Class
    d = Variable
    e = Enumeration
    f = Function
    m = Macro, #define
    s = Structure
    t = Typedef
    u = Union