moo

To ensure the 'longest match' principle (e.g., preventing className from being lexed as class + Name), use the moo.keywords() helper. This helper checks matches against a list of keywords and assigns them a specific type if they match, otherwise falling back to the base type of the rule.

Individual Keyword Types: You can also map specific keywords to unique types by passing an object to moo.keywords() instead of an array.

// Using an array for shared type 'keyword'
moo.compile({
  IDEN: {match: /[a-zA-Z]+/, type: moo.keywords(['while', 'if', 'else'])}
})

// Using an object for individual types
moo.compile({
  name: {match: /[a-zA-Z]+/, type: moo.keywords({
    'kw-class': 'class',
    'kw-def': 'def',
  })}
})

Tokens are defined using regular expressions. When defining rules, keep these behaviors in mind:

• Order matters: Earlier rules take precedence over later ones.
• Non-greedy quantifiers: Use *? instead of * to prevent tokens from consuming more text than intended (e.g., for strings).
• Multiline behavior: The dot /./ does not match newlines. Use [^] if you need to match newlines.
• Excluding characters: Be careful with /[^ ]/ or \s, as they may include newlines depending on your intent.

// Use non-greedy quantifiers to avoid over-matching
let lexer = moo.compile({
  string: /".*?"/,
})

lexer.reset('"foo" "bar"')
lexer.next() // -> { type: 'string', value: 'foo' }

Moo supports multiple states to handle context-sensitive tokenization (like string interpolation). Each state has its own set of rules. You can transition between states using annotations on rules:

• next: 'stateName': Moves to the specified state without changing the stack.
• push: 'stateName': Pushes the current state onto the stack and moves to the new state.
• pop: 1: Removes the top state from the stack and returns to the previous state.

Rules only match if they are defined within the current active state.

let lexer = moo.states({
  main: {
    strstart: {match: '`', push: 'lit'},
    ident:    /\w+/,
  },
  lit: {
    strend:   {match: '`', pop: 1},
    const:    {match: /(?:[^$`]|\$(?!\{))+/, lineBreaks: true},
  },
})

Moo tracks line and column numbers, but for performance reasons, this is disabled by default. To enable line tracking, you must apply the lineBreaks: true option to any rules that might contain newlines (such as a dedicated newline token).

For optimal performance, match newlines in a dedicated token rather than inside every other rule.

let lexer = moo.compile({
  newline: {match: '\n', lineBreaks: true},
  space:  {match: /[ \t]+/},
})

Moo does not support capturing groups in regular expressions. Instead, use the value property in your rule definition to provide a transformation function. This function receives the matched text and returns the processed value.

const lexer = moo.compile({
  string: {match: /"(?:\\\\["\\]|[^\n"\\])*"/, value: s => s.slice(1, -1)},
})

lexer.reset('"test"')
lexer.next() // -> { value: 'test', text: '"test"', ... }

Moo lexers are iterable. You can use a for...of loop to consume tokens or Array.from(lexer) to convert all tokens into an array. For lookahead capabilities, you can use the itt package.

// Standard iteration
for (let here of lexer) {
  // here is a Token object
}

// Convert to array
let tokens = Array.from(lexer);

Calling reset(text) clears the internal buffer and resets line, column, and offset counts.

To resume lexing from a specific point, use lexer.save() to capture the current state (including line/column info) and pass that object as the second argument to reset(text, info).

lexer.reset('some line\n')
let info = lexer.save() // -> { line: 10 }
// ... consume tokens ...
lexer.reset('a different line\n', info)
lexer.next() // -> { line: 10 }

By default, Moo throws an Error if no rules match. You can customize this behavior:

1. Return an error token: Define a rule using moo.error. This token will contain the remainder of the buffer.
2. Partial error matching: Define a rule that matches specific characters but is marked with error: true.
3. Pretty-print errors: Use lexer.formatError(token, "message") to generate a human-readable error string showing the location of the offending token.

// Return an error token instead of throwing
moo.compile({
  myError: moo.error,
})

// Or match specific characters as errors
moo.compile({
  myError: {match: /[\$?`]/, error: true},
})

// Format a pretty error message
throw new Error(lexer.formatError(token, "invalid syntax"))

The next() method returns a Token object with the following properties:

• type: The name of the token group (from compile).
• text: The raw string that was matched.
• value: The string matched, transformed by a value function if provided.
• offset: The byte offset from the start of the buffer.
• lineBreaks: The number of line breaks found in the match (0 if lineBreaks: false).
• line: The starting line number (starting from 1).
• col: The starting column number (starting from 1).

A Lexer instance is iterable. You can use a for...of loop to consume all tokens until EOF.

const tokens = lexer; // Lexer implements Symbol.iterator
for (const token of tokens) {
  console.log(token.type, token.text);
}