SQLGlot

SQLGlot's transpilation process is composed of three primary modules that work in sequence:

1. Tokenizer: Converts raw SQL code into a sequence of tokens (the smallest units of meaningful information like keywords and symbols).
2. Parser: Converts the sequence of tokens into an Abstract Syntax Tree (AST) representing the semantics of the code.
3. Generator: Converts the AST back into SQL code.

Transpilation between different SQL dialects (database-specific syntaxes) is achieved by overriding these three modules for a specific dialect. The base sqlglot dialect provides a common implementation to reduce duplication.

The parser module converts a list of tokens (from the tokenizer) into an Abstract Syntax Tree (AST). The AST captures the semantics (meaning) of a SQL statement rather than just its syntax. This semantic representation is what enables SQLGlot to transpile code between different SQL dialects.

Key concepts:

• Expressions: Each semantic concept in the AST is represented by an expression.
• Recursive Descent: The parser uses mutually recursive _parse_ methods (e.g., _parse_create()) to handle SQL syntax and operator precedence.
• Command Fallback: If SQLGlot cannot parse a specific statement, it falls back to storing the unparsed code in an exp.Command expression. This allows the code to be returned unmodified, though dialect-specific transpilation will not apply to that segment.

SQLGlot processes SQL queries through a series of stages to transform a raw string into executed data. The lifecycle follows these steps:

1. Tokenizing: Converts the SQL string into a list of Token objects, preserving metadata like line/column info and comments.
2. Parsing: A handwritten recursive descent parser converts tokens into an Abstract Syntax Tree (AST).
3. Optimizing: The AST is transformed directly (rather than a logical plan) using a set of rules to create a 'canonical' SQL representation.
4. Planning: The optimized AST is converted into a Directed Acyclic Graph (DAG) logical plan consisting of steps like Scan, Sort, Set, Aggregate, and Join.
5. Executing: The logical plan is iterated to produce results. The Python engine uses a queue to run each step, passing intermediary tables to the next.

SQLGlot uses pdoc to serve its API documentation. You can access a hosted version at sqlglot.com or build and serve it locally using make.

make docs-serve

If you are contributing to SQLGlot or developing with it, you can install the development requirements using a local checkout and make install-dev. You can optionally prefix with UV=1 to use uv for the installation.

# Optionally prefix with UV=1 to use uv for the installation
make install-dev

The Tokenizer (lexical analysis) breaks down SQL code into Token objects. Each token includes its token_type (from the TokenType enum), the raw text, and positional metadata (line, col, start, end) used for error reporting.

SQLGlot uses a TokenType enum to map various lexemes to a single type (e.g., both != and <> map to TokenType.NEQ). The logic is driven by Tokenizer.KEYWORDS and Tokenizer.SINGLE_TOKENS dictionaries.

from sqlglot import tokenize

tokens = tokenize("SELECT b FROM table WHERE c = 1")
for token in tokens:
    print(token)

# Output includes Token objects with metadata:
# <Token token_type: <TokenType.SELECT: 'SELECT'>, text: 'SELECT', line: 1, col: 6, start: 0, end: 5, comments: []>
# <Token token_type: <TokenType.VAR: 'VAR'>, text: 'b', line: 1, col: 8, start: 7, end: 7, comments: []>
# ...

You can install SQLGlot via PyPI. There are two versions available:

1. Pure Python version: Standard installation.
2. C extensions version: Compiled with mypyc. This version is roughly 3-5x faster than the pure Python version. It uses a prebuilt wheel if available for your platform, otherwise it builds from source.

If you are working with a local checkout, you can use make install. You can optionally prefix the command with UV=1 to use uv for the installation.

# Pure python version
pip3 install sqlglot

# C extensions compiled with mypyc
pip3 install "sqlglot[c]"

To trace which table a column belongs to, use sqlglot.optimizer.qualify.qualify. This function prefixes columns with their respective table names (e.g., changing a to x.a).

Note: If the SQL query is ambiguous (e.g., SELECT a FROM x JOIN y), you must provide a schema dictionary to qualify so it can correctly disambiguate the columns.

Once qualified, you can use sqlglot.optimizer.scope.find_all_in_scope to map columns to their source scopes without traversing into subqueries.

If your specific dialect is not supported, you have two options:

1. Subclass an existing dialect.
2. Ship a dialect as a separate package (Dialect Plugin).

Note: Subclassing may not work correctly if the sqlglot[c] (C extensions) version is installed; custom dialects may require the pure Python version.