DOCSEARCHbyAlgolia
Submit Docs

ttp (Template Text Parser)

repository·master·Indexed 19 days ago

https://github.com/dmulyalin/ttp

A Python library for parsing semi-structured text, such as network device CLI outputs, into structured hierarchical data using templates. It dynamically generates regexes from user-defined templates and supports advanced features like XML-like tags (<vars>, <group>, <output>), template composition via the <extend> tag, and a CLI tool for processing data files with various output formats including JSON and YAML.

Tokens
61.1K
Snippets
186
Records
238
Agent score
64%

What's inside ttp

  1. Overview of TTP (Template Text Parser)

    master

    TTP is a Python module designed for fast parsing of semi-structured text data using templates. While primarily developed for programmatic access to CLI output from networking devices, it can be used to parse any semi-structured text that exhibits distinctive repetition patterns.

    In its simplest usage pattern, TTP requires two inputs:

    1. Data to parse: The raw semi-structured text.
    2. Parsing template: A template defining how to extract information.

    It returns a structured representation of the extracted information. A single dataset can be parsed using multiple different templates.

  2. Overview of TTP Output Formatters

    master

    TTP supports various output formatters to transform parsed Python data structures (dictionaries, lists, etc.) into different representations.

    Available formatters:

    • raw: Returns the native Python structure without conversion to string (default).
    • yaml: Transforms results into YAML-structured multi-line text (requires PyYAML).
    • json: Transforms results into JSON-structured multi-line text using json.dumps() with sort_keys=True, indent=4, and separators=(',', ': ').
    • pprint: Uses Python's pprint module for human-readable layout.
    • table: Transforms results into a list of lists (first list is headers, subsequent lists are rows).
    • csv: Emits a CSV spreadsheet using the table formatter logic.
    • tabulate: Emits a plain-text table using the tabulate module.
    • excel: Emits an Excel table using the openpyxl module.
    • jinja2: Renders results using a Jinja2 template provided within the <output> tag.
    • N2G: Produces an XML-structured diagram using the N2G module.

  3. Parse semi-structured text with TTP

    master

    TTP (Template Text Parser) is a Python library designed to transform raw, semi-structured text (like network device CLI outputs) into structured data using templates. It dynamically derives regexes from these templates and allows for complex data transformations, hierarchical structuring, and custom processing via built-in or custom functions.

    from ttp import ttp

data = """
interface Loopback0
 description Router-id-loopback
 ip address 192.168.0.113/24
!"
"""

template = """
interface {{ interface }}\n ip address {{ ip }}/{{ mask }}\n description {{ description }}
"""

parser = ttp(data, template)
parser.parse()
print(parser.result())

  4. Overview of TTP Returners

    master

    Returners in TTP define the destination for processed data. They allow you to emit, save, or return results to different targets. TTP supports four primary returner types:

    • self: The default returner. It returns the processed data back to the calling function, enabling output chaining or programmatic retrieval when TTP is used as a Python module.
    • file: Saves results to a text file on the local file system. One file is produced per template containing all results for all inputs and groups.
    • terminal: Prints results directly to the terminal screen, with support for colorized output.
    • syslog: Sends results to remote Syslog servers over UDP.

  5. Use Lookup Tables to enrich parsing results

    master

    The <lookup> tag defines a lookup table that TTP transforms into a dictionary. You can use this dictionary to look up values and include them in your parsing results using the lookup or rlookup match variable functions.

    Core Attributes

    AttributeDescription
    nameMandatory. The name of the lookup table used to reference it in the lookup function.
    loadThe name of the loader to use (e.g., python, yaml, json, ini, csv). Defaults to python.
    includeAbsolute OS path to a file containing the lookup table data.
    key(For csv loader only) Specifies the column name to use as the dictionary key.
    databaseName of a database loader (e.g., geoip2) to use.

    Usage Pattern

    1. Define the lookup table in your template using <lookup>.
    2. In your parsing logic, use the lookup filter on a variable to retrieve data from that table.
    3. Use add_field="field_name" to nest the looked-up data under a specific key in your result.
    <lookup name="my_table" load="yaml">
key: value
</lookup>

<group name="data">
{{ variable | lookup("my_table", add_field="details") }}
</group>

  7. Chain multiple functions using `functions` or `chain`

    master

    To ensure functions run in a specific, predictable order, use the functions or chain attribute within a <group> tag. These attributes are interchangeable. You can define a pipe-separated string of function calls or reference a template variable that contains a list or pipe-separated string of functions.

    Important: Use the pipe symbol | to separate function names, not a comma.

    Advantages of chaining:

    • Functions execute in the exact order specified.
    • Enables reuse by referencing a single template variable across multiple groups.
    • Improves readability for complex logic.
    <!-- Direct usage in group tag -->
<group name="interfaces_macro" functions="contains('ip') | macro('description_mod') | macro('check_if_svi')">
...
</group>

<!-- Using a template variable (chain) -->
<vars>
chain1 = [
    "del(vlan) | set('set_value', 'set_key')",
    "contains_val(interface, 'Loop')",
    "macro('test_macro')"
]
</vars>

<group chain="chain1">
...
</group>

  8. Advanced TTP Template Syntax

    master

    TTP templates support advanced XML-like tags to control parsing logic:

    • <vars>: Define variables and execution chains (e.g., chain_1 = ["set('var_name')", "lookup(...)"]) to be used during parsing.
    • <group>: Defines a parsing group.
      • name: The name of the group (supports interpolation like bgp_state.{{ peer }}).
      • input: Specifies which input block this group should parse.
      • record: If set, the group results are recorded as a dictionary.
      • chain: Specifies a variable chain to execute for each match.
    • <output>: Defines how results are formatted and returned.
      • name: Identifier for the outputter.
      • format: The output format (e.g., tabulate).
      • path: The path within the result hierarchy to output.
      • returner: The destination (e.g., terminal).
      • headers: Comma-separated list of column headers for table formats.

  9. Configure Group Attributes in TTP

    master

    Group tags (<g>, <grp>, or <group>) can use several attributes to control how data is parsed and structured in the final results. These attributes are provided as strings within the tag.

    AttributeDescription
    nameUniquely identifies the group and specifies the dot-separated path in the results structure.
    inputSpecifies the name of an <input> tag or an OS file path to use as the data source.
    defaultSets a default value for all variables in the group if no matches are found.
    methodDefines the parsing logic: group (default) or table.
    outputA comma-separated list of output tag names to process the group results through, in sequence.

    Note: The input and output attributes are only supported on top-level groups; they are ignored on nested groups.

  10. How TTP handles relative vs absolute paths in group names

    master

    By default, TTP treats the name attribute of a <group> as a relative path. This means the group's position in the resulting JSON hierarchy is determined by expanding its name relative to its parent groups.

    Relative Paths (Default)

    When you define a group name like <group name="VRFs">, TTP builds a nested structure. If VRFs is inside bgp_config, the path becomes [bgp_config, VRFs].

    Absolute Paths

    To flatten the hierarchy and prevent a group from being nested inside its parents, you can specify an absolute path by prepending the name attribute with a forward slash /. This tells TTP to treat the name as a fixed path rather than expanding it relative to the parent.

    Flattening with Anonymous Groups

    You can use name="/" to create an "anonymous" group. This effectively flattens the results by stripping the hierarchical nesting for that specific group, substituting the name with _anonymous_* internally to allow the data to sit at a higher level in the results structure.

    <!-- Relative Path (Default) -->
<group name="bgp_config">
  <group name="VRFs">
    <group name="neighbors">
    </group>
  </group>
</group>

<!-- Absolute Path (Flattens hierarchy) -->
<group name="bgp_config">
  <group name="VRFs">
    <group name="/neighbors">
    </group>
  </group>
</group>

  11. Access TTP internals within Macro Tags via the `_ttp_` dictionary

    master

    When writing Python code inside a <macro> tag, you have access to a special dictionary named _ttp_. This dictionary is injected into the global space of your macro functions and contains references to all:

    • Groups
    • Inputs
    • Outputs
    • Match variables
    • Getter functions

    You can use this _ttp_ dictionary to call TTP's internal functions directly from your custom Python logic.

  12. Nesting groups within an anonymous group

    master

    An anonymous <group> tag (one without a name attribute) supports all standard group attributes, functions, and nesting.

    Important: When nesting groups inside an anonymous group, any child <group> will inherit the name attribute from its parent groups. If the parent is anonymous, the child's naming behavior depends on whether the child itself defines a name.