IWE Documentation

Within a single operation, predicates and actions are processed in a specific sequence. Each step intersects with the results of the previous step:

1. Filter (filter): Narrows the corpus by per-document predicates (frontmatter and graph operators). This applies to all four operations.
2. Search (search): (Only on find operations) Intersects the filtered survivors with search matches and provides default ordering. If search is absent, this step is skipped.
3. Sort (sort): Orders the resulting matched set. On find with search but no explicit sort, the relevance order from the search step is used.
4. Limit (limit): Caps the number of documents in the matched set.
5. Action: The final execution step:
   • find: Projects and returns the matches.
   • count: Returns the integer count of matches.
   • update: Applies update operators atomically and returns the rendered patches.
   • delete: Returns the keys to be removed.

Filters are written in YAML. A document matches if every top-level key matches (implicit AND).

Bare Equality

Matches exact values. For arrays, a bare scalar tests membership.

status: draft
tags: rust

Note: Cross-type comparisons (e.g., string vs integer) are always false.

Operator Expressions

Use $-prefixed keys for complex logic. Operators within a single mapping are ANDed together.

Common operators:

• $gt, $gte, $lt, $lte: Numeric comparisons.
• $in: Matches if value is in the provided list.
• $nin: Matches if value is NOT in the provided list.
• $exists: Checks for field presence (e.g., $exists: true).
• $all: Matches if an array contains all specified elements.
• $size: Matches exact array length or uses count comparisons (e.g., $size: { $gte: 3 }).

# Example operator expressions
priority: { $gt: 3 }
score:    { $gte: 3, $lte: 7 }
status:   { $in: [draft, review] }
stage:    { $nin: [archived, deleted] }
reviewed: { $exists: true }
tags:     { $all: [rust, async] }
labels:   { $size: 0 }
topics:   { $size: { $gte: 3 } }

IWE uses an ordered, sequential, and greedy matching algorithm without backtracking to bind document sections to schema entries. This process happens recursively for the document and every bound section.

Matching Process

1. Sequential Walk: The system walks through the actual sections of a document in order, maintaining a pointer to the current entry in the schema list.
2. Header Matching: For each section, the system looks for the first entry (at or after the current pointer) whose header schema matches the section's header text.
   • An entry with no header defined acts as a wildcard and matches any section.
   • Once a section binds to an entry, the pointer advances to that entry. Entries appearing before the pointer are considered "closed" and cannot be bound again.
3. Additional Sections: If a section does not satisfy any entry at or after the current pointer (including sections that would match a previously closed entry), it is treated as an additionalSection.
4. Validation: After the walk, the system validates:
   • Occurrence Counts: Each entry's total bound count is checked against minContains and maxContains.
   • Content Constraints: Each bound section is validated against maxTokens, maxDepth, allSections, and nested sections requirements.

Critical Rules and Consequences

• Header-Driven Binding: Binding is determined solely by the header. If a section matches a header but fails to contain its required sub-sections, it is still bound to that entry and will report a validation error rather than falling through to additionalSections.
• Total Counts vs. Consecutive Runs: minContains and maxContains apply to the total number of times an entry is bound throughout the document, not just consecutive occurrences. For example, if a schema expects 2 dates and the document contains date, date, other, date, it counts as 3 dates, violating maxContains: 2.
• Wildcard Placement: A headerless (wildcard) entry greedily absorbs all subsequent sections. Therefore, a wildcard must be the last entry in your schema. Placing a wildcard earlier is a schema error.
• No Backtracking: The matching is deterministic. You should always list your schema entries from most specific to least specific to ensure correct binding.

The key_template option supports several variables to control how extracted files are named.

Basic Variables

• {{id}}: A random unique identifier (e.g., 123).
• {{today}}: Current date (uses date_format from [library] section; default %Y-%m-%d).
• {{title}}: The title of the section being extracted (sanitized for filenames).
• {{slug}}: A URL-friendly version of the title (lowercase, alphanumeric, dashes for separators).

Parent Section Variables

• {{parent.title}}: Title of the parent section.
• {{parent.slug}}: URL-friendly version of the parent section title.
• {{parent.key}}: Key of the parent document.

Source Document Variables

• {{source.key}}: Full key of the source document.
• {{source.file}}: Filename portion of the source document key.
• {{source.title}}: Title (first header) of the source document.
• {{source.slug}}: URL-friendly version of the source document title.
• {{source.path}}: Directory path of the source document.

Sections within a document or a parent section are matched against the sections entry list using an ordered, sequential, and greedy algorithm without backtracking.

1. Sequential Walk: The system walks through the actual sections in document order. It maintains a pointer into the schema's entry list.
2. Binding: For each section, the system finds the first entry (at or after the current pointer) whose header schema is satisfied. If an entry has no header defined, it acts as a wildcard and matches any section. Once a section binds to an entry, the pointer advances to that entry. Entries before the pointer are closed.
3. Additional Sections: If a section does not satisfy any entry at or after the pointer, it is considered additional and is handled by the additionalSections policy.
4. Validation: After the walk, the system checks if each entry's bound count satisfies minContains and maxContains. It then recursively validates the bound section against maxTokens, maxDepth, allSections, and nested sections.

Key Rules & Consequences:

• Header-driven: Binding is decided by the header alone. A section that matches a header but is missing required sub-sections will still bind to that entry (and report errors) rather than falling through to additionalSections.
• Greedy Wildcards: A headerless (wildcard) entry greedily absorbs all remaining sections. Therefore, a wildcard must always be the last entry in the list. Placing a wildcard earlier is a schema error.
• No Backtracking: Matching is deterministic. If a section matches an entry that is already 'closed' (before the current pointer), it will not re-bind to it; instead, it will be treated as an additional section.