Doctrine Lexer

A Lexer and a Parser are used in sequence to process source code (like DQL) into an Abstract Syntax Tree (AST):

1. Lexer: Scans the input string and breaks it into a stream of tokens. The parser interacts with the lexer using methods like moveNext() and by inspecting the lookahead property (which contains the current token's type and value).
2. Parser: Consumes the tokens provided by the lexer. It uses the token types (e.g., Lexer::T_SELECT) to decide which grammar rules to apply (e.g., calling SelectStatement()).
3. AST (Abstract Syntax Tree): The final output of the parser. It is a tree representation of the syntactic structure of the source code, where each node represents a construct (like a SelectStatement). This tree can then be used for further transformations, such as converting the language into SQL.

// 1. The Lexer breaks the string into tokens
$lexer = new MyLexer('SELECT u FROM User u');

// 2. The Parser uses the Lexer to build the tree
$parser = new MyParser($lexer);
$AST = $parser->getAST(); // Returns an AST object

To create a custom lexer, extend Doctrine\Common\Lexer\AbstractLexer and implement three required abstract methods. These methods define how the lexer identifies patterns and determines token types.

1. getCatchablePatterns(): Returns an array of regex patterns that represent tokens where the value should be captured.
2. getNonCatchablePatterns(): Returns an array of regex patterns for tokens where the value is not captured.
3. getType(string &$value): Determines the integer token type. This method receives the token value by reference, allowing you to modify or filter the value during the type determination process.

/**
 * Lexical catchable patterns.
 *
 * @return string[]
 */
abstract protected function getCatchablePatterns();

/**
 * Lexical non-catchable patterns.
 *
 * @return string[]
 */
abstract protected function getNonCatchablePatterns();

/** Retrieve token type. Also processes the token value if necessary. */
abstract protected function getType(string &$value): int;

Once a lexer is implemented, you can use it to tokenize a string and iterate through the resulting tokens. The typical workflow is:

1. Call setInput($string) to provide the source text.
2. Call moveNext() to advance the lexer to the first token.
3. Use a loop to check the lookahead property. If lookahead is false, you have reached the end of the input.
4. Inside the loop, call moveNext() to advance to the next token.
5. Access the current token via the token property. You can check the token type using isA(TOKEN_TYPE) and retrieve its content via value.

Note: The first moveNext() call is required to position the lexer at the start of the token stream.

$lexer->setInput($string);
$lexer->moveNext(); // Advance to first token

while (true) {
    if (!$lexer->lookahead) {
        break;
    }

    $lexer->moveNext();

    if ($lexer->token->isA(CharacterTypeLexer::T_UPPER)) {
        $upperCaseChars[] = $lexer->token->value;
    }
}

To create a custom lexer, extend Doctrine\Common\Lexer\AbstractLexer and implement the following protected methods:

1. getCatchablePatterns(): Return an array of regular expressions for patterns that represent meaningful tokens (e.g., identifiers, numbers, strings).
2. getNonCatchablePatterns(): Return an array of regular expressions for patterns that should be ignored (e.g., whitespace) or used to catch single characters.
3. getType(&$value): Implement the logic to map a captured string $value to a specific token type integer.

It is a common pattern to categorize token constants into ranges:

• Tokens < 100: Basic symbols and literals (e.g., T_INTEGER, T_COMMA).
• Tokens >= 100: Identifiers and names (e.g., T_IDENTIFIER, T_ALIASED_NAME).
• Tokens >= 200: Keywords (e.g., T_SELECT, T_WHERE).

use Doctrine\Common\Lexer\AbstractLexer;

class MyLexer extends AbstractLexer
{
    public const T_MY_TOKEN = 200;

    protected function getCatchablePatterns(): array
    {
        return ['[a-z]+'];
    }

    protected function getNonCatchablePatterns(): array
    {
        return ['\s+'];
    }

    protected function getType(&$value): int
    {
        if ($value === 'my_token') {
            return self::T_MY_TOKEN;
        }
        return 1; // T_NONE
    }
}

The Lexer provides a low-level API to traverse an input string and analyze tokens by type, value, and position. Use the following methods to control the lexing process:

Input and Position Management

• setInput($input): Sets the input data to be tokenized. This immediately resets the lexer and tokenizes the new input.
• reset(): Resets the lexer state.
• resetPeek(): Resets the peek pointer to 0.
• resetPosition($position = 0): Resets the lexer position on the input to the specified position.

Token Lookahead and Validation

• isNextToken($token): Checks if the current lookahead matches the specified token.
• isNextTokenAny(array $tokens): Checks if any of the provided tokens match the current lookahead.
• isA($value, $token): Checks if the given value is identical to the specified token.
• peek(): Moves the lookahead token forward.
• glimpse(): Peeks at the next token, returns it, and immediately resets the peek pointer.

Traversal

• moveNext(): Moves to the next token in the input string.
• skipUntil($type): Skips input tokens until a token of the specified type is encountered.