Obscenity

Obscenity's documentation is split into two main parts:

1. Reference Documentation: Automatically generated using TypeDoc, providing detailed technical specifications for the API.
2. Guides: Hand-written articles providing conceptual explanations and usage instructions.

The Matcher interface is responsible for searching for blacklisted terms within a given text string. A key feature of the Matcher is that it ignores parts of the text that are matched by whitelisted terms, preventing false positives.

Common implementations include RegExpMatcher (which uses regular expressions).

The RegExpMatcher is the primary implementation of the Matcher interface in Obscenity. It uses regular expressions and string searching to identify blacklisted terms while respecting whitelisted terms.

Its internal logic follows these steps:

1. Apply whitelistMatcherTransformers to the text.
2. Find all whitelisted terms in the text.
3. Apply blacklistMatcherTransformers to the text.
4. For each blacklisted term pattern:
   • Find all matches in the text.
   • For each match, check if it overlaps with a whitelisted term.
   • If no whitelisted term covers that part of the text, emit a match.

The CharacterMapping type is used to define how characters should be transformed. It can be implemented as either a Map<string, string> or a Record<string, string>.

Important Logic Rule: The mapping is inverted compared to a standard replacement dictionary. The key of the map/object must be the target (transformed) character, and the value must be a collection of characters (as a string) that should be mapped to that target character.

// Example of a CharacterMapping where 'a' is the target
// and 'á', 'à', 'â' are the source characters mapping to it.
const mapping: CharacterMapping = {
  'a': 'áàâ'
};

// Or using a Map
const mappingMap: CharacterMapping = new Map<string, string>([
  ['a', 'áàâ']
]);

The whitelistedTerms property allows you to define strings that should exempt parts of the input text from being flagged by the blacklist. If a whitelisted term matches a segment of text, any blacklisted pattern found within that specific segment will not be reported.

Example Scenario: If you have a blacklisted pattern for penis but you want to allow the phrase pen is (as in "the pen is mightier"), adding pen is to whitelistedTerms will prevent the matcher from flagging that specific instance.

You can use blacklistMatcherTransformers and whitelistMatcherTransformers to modify the input text before the matching engine runs. This is useful for normalizing text (e.g., removing special characters or handling casing) specifically for one side of the matching logic.

• blacklistMatcherTransformers: Applied before blacklisted patterns are matched. Does not affect whitelisted matching.
• whitelistMatcherTransformers: Applied before whitelisted terms are matched. Does not affect blacklisted matching.

Transformers are applied in the order they are provided in the array.

The DataSet<MetadataType> class is used to hold groups of patterns and whitelisted terms. It allows you to optionally associate custom metadata with each phrase. You can build a dataset by adding individual phrases via a builder function or by merging existing datasets. Once configured, the dataset can be transformed into a format compatible with RegExpMatcher or used to enrich match results with the metadata you've attached.

const data = new DataSet<{ originalWord: string }>()
	.addPhrase((phrase) => phrase.setMetadata({ originalWord: 'fuck' })
		.addPattern(pattern`fuck`)
		.addPattern(pattern`f[?]ck`)
		.addWhitelistedTerm('Afck'))
	.build();

In obscenity, a Node is a type alias representing the fundamental building blocks of a pattern. A Node can be any one of three specific types:

1. LiteralNode: Represents a specific, exact sequence of characters.
2. OptionalNode: Represents a part of a pattern that may or may not be present.
3. WildcardNode: Represents a placeholder that matches a range of characters.

When constructing or interacting with patterns, you will encounter these nodes as the constituent parts of the pattern structure.

type Node = LiteralNode | OptionalNode | WildcardNode;

DataSet allows you to group blacklisted and whitelisted terms into "phrases" and attach arbitrary metadata to those groups. This is useful for mapping matched patterns back to original words or categories without using complex conditional logic.

You can define a DataSet using a fluent API, specifying metadata via .setMetadata() and patterns via .addPattern() or whitelisted terms via .addWhitelistedTerm().

import { DataSet, pattern } from 'obscenity';

const dataset = new DataSet<{ originalWord: string }>()
	.addPhrase((phrase) =>
		phrase
			.setMetadata({ originalWord: 'fuck' })
			.addPattern(pattern`fck`)
			.addPattern(pattern`fuck"),
	.addPhrase((phrase) =>
		phrase
			.setMetadata({ originalWord: 'bitch' })
			.addPattern(pattern`bish`)
			.addPattern(pattern`bitch`)
			.addWhitelistedTerm('abish");

Transformers are used to normalize text before it is passed to a matcher. They allow you to resolve variations in text that might be used to bypass filters. Common use cases include:

• Confusable character resolution: Converting look-alike Unicode characters to standard Latin (e.g., Ἢἕļľᦞ -> hello).
• Leet-speak resolution: Converting numbers/symbols to letters (e.g., h3llo -> hello).
• Duplicate character collapsing: Reducing repeated characters (e.g., heeello -> hello).
• Character removal: Ignoring specific characters like spaces or punctuation (e.g., f u c k -> fuck).

The TransformerContainer is a union type that represents all possible transformer container types in the obscenity library. It allows you to work with transformers regardless of whether they are stateless or stateful. It is composed of two primary interfaces:

1. SimpleTransformerContainer: For stateless transformations.
2. StatefulTransformerContainer: For transformations that maintain internal state.

When writing functions that accept a transformer, using TransformerContainer as the type allows your code to be compatible with both simple and stateful implementations.