Grammar-Kit

Grammar-Kit uses specific attributes to help the parser recover from syntax errors in the input stream.

pin

The pin attribute defines a point in a rule that, if reached, marks the match as successful even if subsequent parts of the rule fail. This is used in extendedPin mode (enabled by default) to allow the parser to continue matching the rest of a sequence despite failures.

Example: paren_expr ::= '(' expr ')' {pin=1} means paren_expr is considered successful if at least the ( is matched.

recoverWhile

The recoverWhile attribute tells the parser to skip tokens as long as a specific recovery rule matches. The recovery rule is typically a predicate (often a NOT predicate) and does not consume input itself.

Example:

property ::= id '=' expr {pin=2 recoverWhile=rule_recover}
private rule_recover ::= !(';' | id '=')

In this case, if the property match fails after the = part, the parser will skip all tokens until it encounters either a ; or the start of a new property (id '=').

Modifiers change how rules affect the resulting PSI tree or how the parser behaves. Rules are public by default.

Combinations:

• private left is equivalent to private left inner.
• fake should not be combined with private.

To avoid deep stacks in recursive descent parsers for expressions, use the priority-based compact syntax.

Best Practices:

1. Root Rule: All expression rules should extend a single "root expression rule" using the extends attribute. This collapses redundant nodes in the AST.
2. Priority: Priority increases from top to bottom in the BNF.
3. Associativity: Use the rightAssociative=true attribute when left associativity is not desired.
4. Grouping: Use private rules to group operators with the same priority level.
5. Left Recursion: Use left recursion for binary and postfix expressions.

Example structure:

{
  extends(".*expr")=expr
  tokens=[number="regexp:[0-9]+" id="regexp:[a-z][a-z0-9]*"]
}

// Root rule
expr ::= assign_expr | add_group | mul_group | ... | primary_group

// Priority groups
private mul_group ::= mul_expr | div_expr
private add_group ::= plus_expr | minus_expr

// Public rules
assign_expr ::= expr '=' expr { rightAssociative=true }
div_expr ::= expr '/' expr

{
  extends(".*expr")=expr
  tokens=[number="regexp:[0-9]+" id="regexp:[a-z][a-z0-9]*"]
}

expr ::= assign_expr
  | add_group
  | mul_group
  | unary_group
  | exp_expr
  | qualification_expr
  | primary_group

private unary_group ::= unary_plus_expr | unary_min_expr
private mul_group ::= mul_expr | div_expr
private add_group ::= plus_expr | minus_expr
private primary_group ::= simple_ref_expr | literal_expr | paren_expr

assign_expr ::= expr '=' expr { rightAssociative=true }
unary_min_expr ::= '-' expr
unary_plus_expr ::= '+' expr
div_expr ::= expr '/' expr
expr '*' expr
exp_expr ::= expr ('^' expr) +
paren_expr ::= '(' expr ')'

Grammar-Kit uses a syntax based on Parsing Expression Grammar (PEG).

Key Syntax Elements:

• ::=: The assignment/definition symbol.
• [ ... ]: Optional sequences.
• { | | }: Choices.
• *: Repetition.
• +: One or more.
• ?: Optionality.
• &required !forbidden: Predicate expressions.

Attributes: Attributes are defined as name=value pairs in braces {}. They can be Global (at the top of the file or separated by a semicolon) or Rule-specific (placed immediately after a rule definition).

Example Grammar:

{ generate=[psi="no"] } // Global attribute

private left rule_with_modifier ::= '+' 
left rule_with_attributes ::= '?' {elementType=rule_D} // Rule attribute

private meta list ::= <<p>> (',' <<p>>) * // Meta rule
private list_usage ::= <<list rule_D>> // Meta rule application

// Basic PEG BNF syntax
root_rule ::= rule_A rule_B rule_C rule_D                // sequence expression
rule_A ::= token | 'or_text' | "another_one"             // choice expression
rule_B ::= [ optional_token ] and_another_one?           // optional expression
rule_C ::= &required !forbidden                          // predicate expression
rule_D ::= { can_use_braces + (and_parens) * }           // grouping and repetition

// Grammar-Kit BNF syntax
{ generate=[psi="no"] }                                  // top-level global attributes
private left rule_with_modifier ::= '+'                  // rule modifiers
left rule_with_attributes ::= '?' {elementType=rule_D}   // rule attributes

private meta list ::= <<p>> (',' <<p>>) *                // meta rule with parameters
private list_usage ::= <<list rule_D>>                   // meta rule application

Live Preview allows you to test your BNF grammar against sample text in real-time without generating code or running tests.

How to use Live Preview

1. Open a new file or use an existing one.
2. Invoke the Live Preview action via the context menu or use the shortcut Ctrl+Alt+P (or Meta+Alt+P).
3. Paste your sample text (the input you want to test) into the preview window.
4. Use the following tools to observe the results:
   • Structure toolwindow and File Structure popup (Ctrl+F12 / Meta+F12): To observe the PSI tree.
   • PSI Viewer dialog: To inspect the parsed structure.
   • Start/Stop Grammar Highlighting (Ctrl+Alt+F7 / Meta+Alt+F7): To highlight grammar expressions at the current caret position in a preview editor.

Recommended Workflow

1. Prototype: Use Live Preview to refine the .bnf logic.
2. Generate Lexer: Generate the initial *.flex file from the editor context menu, then generate the *.java lexer from the .flex file.
3. Setup: Create the ParserDefinition and implement parser/lexer tests.
4. Production: Refine the *.flex and *.bnf files separately in your production environment.

To create a new language support in IntelliJ IDEA using Grammar-Kit, follow these steps:

1. Create a grammar file: Create a *.bnf file (see grammars/Grammar.bnf in the plugin for a reference).
2. Tune the grammar: Use the Live Preview mode and the Structure view (Ctrl-Alt-P / Cmd-Alt-P) to refine your rules.
3. Generate code: Use the generator (Ctrl-Shift-G / Cmd-Shift-G) to produce the parser, ElementTypes, and PSI classes.
4. Generate lexer: Create a *.flex file, then use the context menu to generate the lexer and run the JFlex generator.
5. Integrate with IntelliJ: Implement ParserDefinition and register it in your plugin's plugin.xml.
6. Extend functionality: Implement resolve and other non-trivial logic within your PSI classes.

Note for Kotlin Multiplatform: If using the syntax-api, add generate=[parser-api="syntax"] to the grammar header. This generates the parser as a Kotlin object, though PSI classes remain Java.

{ generate=[parser-api="syntax"] }
// Your grammar here

Grammar-Kit can generate Kotlin parsers compatible with the IntelliJ Platform syntax-api, enabling Kotlin Multiplatform (KMP) support.

Activation: Add parser-api="syntax" to the generate attribute in the grammar header.

Key Attributes for Kotlin Generation:

• syntaxElementTypeHolderClass: FQN of the Kotlin object holding SyntaxElementType constants (default: "generated.GeneratedSyntaxElementTypes").
• syntaxElementTypeFactory: FQN of the factory method for custom SyntaxElementType instances.
• syntaxParserUtilObject: FQN of the parser utility object (replaces GeneratedParserUtilBase).
• psiOutputPath: Path to a separate directory for PSI generation (useful for KMP where parser and PSI live in different modules).

Example Configuration:

{
  generate=[parser-api="syntax"]
  parserClass="org.intellij.grammar.expression.ExpressionParser"
  syntaxElementTypeHolderClass="org.intellij.grammar.expression.ExpressionSyntaxTypes"
  syntaxElementTypeFactory="org.intellij.grammar.expression.ExpressionParserDefinition.createSyntaxType"
  syntaxParserUtilObject="com.intellij.platform.syntax.util.runtime.SyntaxGeneratedParserRuntime"
  psiOutputPath="jvm-module/gen"
}

root ::= item *
item ::= id '=' expr

{
  generate=[parser-api="syntax"]
  parserClass="org.intellij.grammar.expression.ExpressionParser"

  syntaxElementTypeHolderClass="org.intellij.grammar.expression.ExpressionSyntaxTypes"
  syntaxElementTypeFactory="org.intellij.grammar.expression.ExpressionParserDefinition.createSyntaxType"
  syntaxParserUtilObject="com.intellij.platform.syntax.util.runtime.SyntaxGeneratedParserRuntime"

  elementTypeHolderClass="org.intellij.grammar.expression.ExpressionTypes"
  elementTypeFactory="org.intellij.grammar.expression.ExpressionParserDefinition.createType"

  psiOutputPath="jvm-module/gen"
}