CFFI (C Foreign Function Interface for Python)

CFFI provides three main ways to expose C libraries to Python, ranging in complexity and performance:

1. In-line, ABI mode: The simplest method. You declare types and load the library directly in your main script using ffi.dlopen(). Best for quick tasks or when you don't want a build step.
2. Out-of-line, ABI mode: Better for organization and faster import times. You use a separate build script to generate a Python module. The module contains the ffi object, but you still use ffi.dlopen() to load the actual library.
3. Out-of-line, API mode: The most powerful and fastest method. It compiles a C extension module. This allows you to access the library at the C level rather than the binary level and provides the most flexibility.

import cffi

# 1. In-line ABI mode
ffi = cffi.FFI()
ffi.cdef("C-like declarations")
lib = ffi.dlopen("libpath")

# 2. Out-of-line ABI mode (in a build script)
ffibuilder = cffi.FFI()
ffibuilder.set_source("package._foo", None)
ffibuilder.cdef("C-like declarations")
if __name__ == '__main__':
    ffibuilder.compile()

# 3. Out-of-line API mode (in a build script)
ffibuilder = cffi.FFI()
ffibuilder.set_source("package._foo", r"""real C code""")
ffibuilder.cdef("C-like declarations with '...' ")
if __name__ == '__main__':
    ffibuilder.compile(verbose=True)

CFFI interacts with Python's threading model in specific ways that require careful handling of shared C data:

1. GIL Behavior: On GIL-enabled builds, CFFI releases the GIL before calling into a C library. This allows for multithreaded speedups but means the GIL does not protect multithreaded shared use of C data structures.
2. Thread-Safe C Libraries: If the underlying C library is thread-safe, no additional locking is required in Python.
3. Non-Thread-Safe C Libraries: If the C library relies on global state or is not re-entrant, you must implement locking in Python (e.g., using threading.Lock) to ensure only one thread accesses the library at a time.
4. Locking Strategies:
   • Global Lock: Use a single threading.Lock if the library is not re-entrant and uses global state.
   • Per-Object Lock: If the library is re-entrant but not thread-safe (i.e., threads can run simultaneously as long as they don't mutate the same object), use a lock associated with specific C objects.
   • Non-blocking Acquire: To avoid deadlocks, use l.acquire(blocking=False) to immediately raise an error if a lock is already held.

import threading

l = threading.Lock()

def work():
    # Protect unsafe C calls with a lock
    l.acquire()
    try:
        lib.increment()
    finally:
        l.release()

# Or using a context manager (preferred)
def work_safe():
    with l:
        lib.increment()

When using C compiler (API mode), you can use the ellipsis ... syntax in cdef() to allow the C compiler to resolve specific details like field offsets, exact integer sizes, or array lengths. This is useful when you don't want to manually specify every detail of a C structure or type.

Supported ellipsis usages:

• Partial Structures/Unions: Use struct { ...; } or union { ...; }. The ...; as the last field allows fields to be undeclared, out of order, or use non-standard alignment. Note: You can only access fields you explicitly declare.
• Integer Types: Use typedef int... foo_t; (or long..., unsigned long long..., etc.). The compiler determines the exact size and signedness. This maps to a standard Python integer type.
• Floating-Point Types: Use typedef float... foo_t; or typedef double... foo_t;. The compiler determines if it is a float or double. Note: long double is not supported via this syntax and must be declared explicitly.
• Opaque Types: Use typedef ... foo_t; to declare a type as opaque. This is useful for pointers where you don't need to inspect the internal structure. This only works for struct-like types, not primitive integers.
• Arrays: Use extern int n[...]; for arrays with unspecified lengths. This supports multidimensional arrays like extern int n[...][...];. In version 1.2+, you can use extern int m[][...]; where the outermost dimension is unknown to the compiler but inner dimensions are known.
• Enums: Use enum foo { A, B, C, ... }; to let the compiler determine the exact values and order of constants.
• Macros/Constants: Use #define FOO ... in cdef() for integer constants. Alternatively, use static const int FOO; for more general types.

Limitations

• Misdeclaring simple integer/float function arguments (e.g., declaring long as int) works because the C compiler performs the cast, but this does not work for function pointers or variadic functions.
• You cannot misdeclare complex types like pointers (e.g., int * as long *) or global arrays.

// Example of partial struct
struct { 
    int a; 
    ...; 
};

// Example of opaque type
typedef ... foo_t;

// Example of integer type detection
typedef int... foo_t;

// Example of array with compiler-filled length
extern int n[...];

CFFI is designed to call existing C libraries from Python based on several core principles:

• Minimal Learning Curve: The interface is based on LuaJIT's FFI. It aims to allow users to call C code using only knowledge of C and Python, avoiding the need for domain-specific languages like Cython or SWIG.
• Python-Centric Logic: Most logic is kept in Python to minimize the amount of C code a user needs to write compared to native C extensions.
• API vs. ABI approach:
  • API Level (Preferred): You write C declarations, and the C compiler is called to validate and link to those constructs. This is the recommended way to work.
  • ABI Level: You can work at the Application Binary Interface level (similar to ctypes), but this is less robust on non-Windows platforms where C libraries may have a specified API but not a stable ABI.
• C Support: CFFI aims for completeness, supporting C89 constructs and macros. Note that some C99 constructs may not be supported. If a macro is complex or 'abused', you can manually wrap it in a saner C function.
• No C++ Support: CFFI does not support C++. To use C++ code, you should write a C wrapper around the C++ code and then call that C API via CFFI.
• Not for Embedding C: CFFI is intended for calling existing C libraries, not for embedding executable C code within Python (unlike Weave).

When using recompile() to produce a CPython C extension module, the module contains internal structures used to lazily initialize types and functions.

Type Descriptors

Types are stored in the global array _cffi_types[]. Initially, entries are odd integers acting as opcodes. At runtime, these are converted into CTypeDescrObject * instances.

Function Globals

Functions are listed in the _cffi_globals array. This is a sorted array of entries that are lazily converted into real Python builtin function objects. Each entry points to an index in _cffi_types[] using the OP_FUNCTION opcode to describe the function's signature (return type and arguments).

Opcode Mechanism

Opcodes are integers where the lowest byte contains type information and the remaining high bytes contain an argument (arg) relevant to that type. Common opcodes used within _cffi_types include:

• OP_PRIMITIVE: The arg is an index in a primitive type list.
• OP_POINTER: The arg is the index of the item type in _cffi_types.
• OP_ARRAY: The arg is the index of the item type in _cffi_types, followed by an opcode containing the array length.
• OP_OPEN_ARRAY: Similar to OP_ARRAY but for unsized arrays (e.g., int[]).
• OP_STRUCT_UNION: The arg is the index in _cffi_structs_unions.
• OP_ENUM: The arg is the index in _cffi_enums.
• OP_TYPENAME: The arg is the index in _cffi_typenames.
• OP_FUNCTION: The arg is the index of the result type in _cffi_types, followed by argument opcodes, terminated by OP_FUNCTION_END.
• OP_FUNCTION_END: The arg's lowest bit indicates if a variadic argument (...) is present.
• OP_NOOP: A simple indirection where the arg is the index to look further in.

In CFFI, a function argument declared as item * is treated identically to item[]. This means you can pass Python lists or byte strings to these arguments.

Important Notes:

• Single items: If you want to pass a single item to a pointer argument, you must wrap it in a list of length 1 (e.g., [[x, y]] for a struct point_s *).
• Lifespan: When passing a Python object (like a byte string) to a char * argument, the CFFI creates the required C data structure just before the call and frees it immediately after. The pointer is only valid for the duration of the call.
• Persistent Pointers: If the C function needs to store the pointer for later use, do not pass a Python byte string directly. Instead, create a persistent buffer using ffi.new("char[]", mystring) or ffi.from_buffer() with a bytearray.

# Passing a single struct to a pointer argument
# If the C function is: void process_point(struct point_s *p);
lib.process_point([{'x': 5, 'y': 10}])

# Creating a persistent buffer for a char pointer
p = ffi.new("char[]", mystring)  # This makes a copy that lives as long as 'p' exists