SLJIT Documentation

SLJIT is a stack-less, low-level, and platform-independent JIT compiler that functions similarly to a platform-independent assembler. It uses a Low-level Intermediate Representation (LIR) to allow developers to generate machine code for various CPU architectures with performance close to native assembly.

Key characteristics:

• Low-level control: Provides direct control over generated machine code, including direct access to integer and floating-point registers.
• No automatic register allocation: It does not manage registers for you; it is designed to be a backend for other JIT compiler libraries.
• Portability: By targeting a platform-independent LIR, code can be compiled to multiple architectures.
• All-in-one compilation: Supports a mode where the SLJIT API can be completely hidden from external use.
• Serialization: Supports serializing the compiler into a byte buffer, enabling Ahead-of-Time (AOT) compilation or resuming code generation after deserialization (partial AOT).

SLJIT uses a Low-level Intermediate Representation (LIR) that abstracts away specific CPU details while providing optimization opportunities. The model consists of:

• Integer Registers: Can store int32_t (4 byte) or intptr_t (4 or 8 byte) values.
• Floating Point Registers: Can store single (4 byte) or double (8 byte) precision values.
• Boolean Status Flags: Used for conditional branching. To maintain compatibility across architectures, SLJIT primarily exposes a zero (equal) flag and a variable flag (which maps to flags like CARRY or OVERFLOW depending on the operation).
• Vector Registers: Supported on some platforms, often aliasing floating point registers.

Crucial Rule for Registers: When an instruction uses a register as a source operand, the data type of the register must match the data type expected by the instruction. Mismatched types result in undefined behavior or crashes (e.g., on MIPS-64). However, you can always overwrite a register with a new type, which discards the previous value.

// VALID: int32_t loaded into SLJIT_R0, then byte swapped as int32_t
sljit_emit_op1(compiler, SLJIT_MOV32, SLJIT_R0, 0, SLJIT_MEM1(SLJIT_R1), 0);
sljit_emit_op1(compiler, SLJIT_REV32, SLJIT_R0, 0, SLJIT_R0, 0);

// INVALID: intptr_t loaded into SLJIT_R0, then treated as int32_t
sljit_emit_op1(compiler, SLJIT_MOV, SLJIT_R0, 0, SLJIT_MEM1(SLJIT_R1), 0);
sljit_emit_op1(compiler, SLJIT_REV32, SLJIT_R0, 0, SLJIT_R0, 0); // Undefined behavior

// VALID: Overwriting a register with a different type
sljit_emit_op1(compiler, SLJIT_MOV, SLJIT_R0, 0, SLJIT_MEM1(SLJIT_R1), 0); // R0 is intptr_t
sljit_emit_op1(compiler, SLJIT_MOV32, SLJIT_R0, 0, SLJIT_MEM1(SLJIT_R2), 0); // R0 is now int32_t

The PCRE2 JIT compiler translates regular expression byte code into machine code specialized for the execution order of sub-patterns. For every sub-pattern, the compiler generates two distinct paths:

1. Matching Path: Follows the original concatenation order of the pattern. When a sub-pattern matches successfully, the engine proceeds to the next matching path.
2. Backtracking Path: Generated in reversed concatenation order. If a matching path fails, the engine falls back to the previous sub-pattern via the backtracking path.

Because of this specific generation order, horizontal control transfers between adjacent paths often do not require explicit jump instructions. The current position in the input is tracked using a STRING_POINTER variable. While this variable is valid during matching paths, it may be undefined during backtracking paths, requiring the backtracking path to restore or set it before jumping back to a matching path.

     # Start of matching paths
   ENTER
     # Matching path of matching the "a" letter
   MATCH "a", IF FAILS GOTO L6
     # Matching path of (?:)+
   STACK_PUSH NULL
L1:
     # Matching path of matching the \w special character
   MATCH WORD_CHARACTER, IF FAILS GOTO L5
     # Matching path of matching the \d special character
   MATCH DIGIT_CHARACTER, IF FAILS GOTO L4
     # Continue the matching path of (?:)+
   STACK_PUSH STRING_POINTER
   GOTO L1
L2:
     # Matching path of matching the "d" letter
   MATCH "d", IF FAILS GOTO L3
   RETURN SUCCESS

     # Start of backtracking paths
     # Backtracking path of matching the "d" letter (empty)
L3:
     # Backtracking path of (?:)+ (empty)
     # Backtracking path of matching the dot special character (empty)
L4:
     # Backtracking path of matching the "b" letter (empty)
L5:
     # Continue backtracking path of (?:)+
   STRING_POINTER = STACK_POP
   IF STRING_POINTER != NULL GOTO L2
     # Backtracking path of matching the "a" letter (empty)
L6:
   RETURN FAIL

Branching in SLJIT allows you to divert control flow to implement high-level constructs like conditionals and loops. There are two types of branches (jumps):

1. Conditional: Only taken if a specific condition is met; otherwise, execution continues to the next instruction.
2. Unconditional: Always taken.

To implement branching, you use a two-step process involving struct sljit_jump and struct sljit_label:

• Emit a Jump: Use sljit_emit_cmp (conditional) or sljit_emit_jump (unconditional). These return a pointer to a struct sljit_jump.
• Emit a Label: Use sljit_emit_label to create a target location, which returns a struct sljit_label.
• Connect them: Use sljit_set_label(jump, label) to link the jump to its target label. This mechanism is conceptually similar to goto and labels in C.

/* Conceptual workflow */
struct sljit_jump *j = sljit_emit_cmp(C, SLJIT_EQUAL, ...);
struct sljit_label *l = sljit_emit_label(C);
sljit_set_label(j, l); // Connect jump to label

SLJIT supports a wide range of target architectures and low-level operations:

Supported Architectures

• x86 (32 / 64-bit)
• ARM (32 / 64-bit)
• RISC-V (32 / 64-bit)
• s390x (64-bit)
• PowerPC (32 / 64-bit)
• LoongArch (64-bit)
• MIPS (32 / 64-bit)

Key Capabilities

• Low-level Control: Direct access to integer and floating-point registers, and support for stack space allocation for local variables.
• Advanced Operations: Supports self-modifying code, tail calls, fast calls, endianness switching (byte order reverse), unaligned memory accesses, SIMD, and atomic operations.
• Compilation Modes:
  • All-in-one compilation: Allows the SLJIT API to be completely hidden from external use.
  • Serialization: The compiler can be serialized into a byte buffer. This enables Ahead-of-Time (AOT) compilation or partial AOT compilation where code generation can be resumed after deserialization.

SLJIT code generation follows a specific lifecycle to ensure interoperability with standard C functions (adhering to the system's ABI, such as System V or Windows x64):

1. Create a compiler: Initialize a compiler context using sljit_create_compiler.
2. Emit function prologue: Use sljit_emit_enter to establish a call frame, save necessary registers, and allocate stack space for local variables.
3. Emit operations: Use sljit_emit_opN functions to generate machine instructions using registers and operands.
4. Emit return: Use sljit_emit_return to move the result to the return register and exit the function.
5. Generate code: Call sljit_generate_code to produce the executable machine code.
6. Execute: Cast the resulting pointer to a compatible function pointer type.
7. Cleanup: Free the compiler context with sljit_free_compiler and the generated code with sljit_free_code.

struct sljit_compiler *C = sljit_create_compiler(NULL);

sljit_emit_enter(C, 0, SLJIT_ARGS3(W, W, W, W), 1, 3, 0);
// ... emit operations ...
sljit_emit_return(C, SLJIT_MOV, SLJIT_R0, 0);

void *code = sljit_generate_code(C, 0, NULL);
sljit_size_t len = sljit_get_generated_code_size(C);

// Execute
typedef void (*func_t)(void);
func_t func = (func_t)code;
func();

// Cleanup
sljit_free_compiler(C);
sljit_free_code(code, NULL);

Character class checks are optimized using several techniques:

• Caseless Comparison: If there is only a single bit difference between the lower and upper case of a character (e.g., ASCII x and X differ only by bit 6), the JIT uses a single bitwise instruction (chr|0x20)=='x' instead of two separate comparisons.
• UTF-8 Range Specialization: If a character class is limited to characters < 128 in UTF-8 mode, the JIT generates specialized readers that only read the next byte instead of performing a full UTF-8 decode.
• Boundary Check Elimination: When invalid UTF-8 parsing is enabled, the reader checks if at least four bytes remain in the buffer. If so, it proceeds with decoding without further boundary checks, as four bytes is the maximum length of a UTF-8 character.
• Lightweight Helper Functions: For complex UTF decoding, the JIT generates extremely lightweight helper functions that do not set up a full call frame or save CPU registers, behaving similarly to inlined functions.