theseer/tokenizer

Add Tokenizer as a local, per-project dependency using Composer:

composer require theseer/tokenizer

If you only require the library for development purposes (e.g., running test suites), install it as a development-time dependency:

composer require --dev theseer/tokenizer

To convert PHP source code into an XML representation, use the TheSeer\Tokenizer\Tokenizer class to parse the file content into tokens, and then use TheSeer\Tokenizer\XMLSerializer to transform those tokens into an XML string.

The resulting XML structure organizes tokens within <line> elements inside a <source> root element, with each token identified by a name attribute.

$tokenizer = new TheSeer\Tokenizer\Tokenizer();
$tokens = $tokenizer->parse(file_get_contents(__DIR__ . '/src/XMLSerializer.php'));

$serializer = new TheSeer\Tokenizer\XMLSerializer();
$xml = $serializer->toXML($tokens);

echo $xml;

The TokenCollection class is a container for Token objects. It implements IteratorAggregate, ArrayAccess, and Countable, allowing you to treat the collection like an array. You can iterate over it, count the tokens, or access specific tokens by their integer offset.

Note that when using array access (offsetSet), the offset must be an integer and the value must be an instance of TheSeer okenizer\​Token. Failure to meet these requirements will throw a TokenCollectionException.

Sets a Token at a specific integer offset using array syntax (e.g., $collection[$i] = $token).

Throws TokenCollectionException if:

• The offset is not an integer.
• The value is not an instance of TheSeer\tokenizer\u200bToken.

Retrieves a Token at the specified integer offset.

Throws TokenCollectionException if the offset does not exist.