sensitive-word

The sensitive-word library provides several text processing and filtering capabilities:

• stop-word: Filtering sensitive words.
• 拼音 (Pinyin): Pinyin conversion/processing.
• 繁简体转换 (Traditional/Simplified Chinese conversion): Converting between traditional and simplified Chinese characters.
• 全角半角转换 (Full-width/Half-width conversion): Converting between full-width and half-width character sets.
• 重复词 (Duplicate words): Handling or detecting repeated words.

Additional/Planned features:

• 中文英文转换 (Chinese-to-English conversion - TBD)
• 手写 Regex (Custom Regex support)
• 分词 (Word segmentation/Tokenization)

The project uses sensitive-word-data for its built-in dictionaries. If you want to avoid including these in your package (e.g., for Android apps) or if you want to load them from a remote server/encrypted source, exclude the dependency in your pom.xml.

<dependency>
    <groupId>com.github.houbb</groupId>
    <artifactId>sensitive-word</artifactId>
    <version>${sensitive-word.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.github.houbb</groupId>
            <artifactId>sensitive-word-data</artifactId>
        </exclusion>
    </exclusions>
</dependency>

To use the sensitive-word library in your Java project, add the following dependency to your pom.xml file. Ensure you are using JDK 1.8+ and Maven 3.x+.

<dependency>
    <groupId>com.github.houbb</groupId>
    <artifactId>sensitive-word</artifactId>
    <version>0.29.5</version>
</dependency>

The library provides several features to increase hit rates by ignoring common bypass techniques. These can be enabled via SensitiveWordBs using a fluent API:

• ignoreCase(true): Ignore uppercase/lowercase differences.
• ignoreWidth(true): Ignore full-width/half-width character differences.
• ignoreNumStyle(true): Ignore various numeric representations (e.g., circled numbers, superscripts).
• ignoreChineseStyle(true): Ignore Traditional vs Simplified Chinese differences.
• ignoreEnglishStyle(true): Ignore stylized English characters (e.g., mathematical alphanumeric symbols).
• ignoreRepeat(true): Ignore repeated characters used to bypass detection.

// Example: Ignore repeated characters
List<String> wordList = SensitiveWordBs.newInstance()
        .ignoreRepeat(true)
        .init()
        .findAll(text);

Starting from v0.23.0, you can categorize sensitive words with tags and filter results so that only words belonging to specific tags are returned. To use this, implement AbstractWordTag and provide it via .wordTag(new YourTagImplementation()).

// 1. Define your tag implementation
public class MyWordTag extends AbstractWordTag {
    @Override
    protected Set<String> doGetTag(String word) {
        // Return tags associated with the word (e.g., "色情", "广告")
        return dataMap.get(word);
    }
}

// 2. Use it in the builder
SensitiveWordBs sensitiveWordBs = SensitiveWordBs.newInstance()
        .wordDeny(new IWordDeny() { ... })
        .wordTag(new MyWordTag())
        .wordResultCondition(WordResultConditions.wordTags(Arrays.asList("色情")))
        .init();

By default, specific pattern detections like Email, URL, and IPv4 are disabled. You can enable them using the SensitiveWordBs builder:

• enableEmailCheck(true): Detects email addresses.
• enableUrlCheck(true): Detects URLs. You can further customize this with .wordCheckUrl(WordChecks.urlNoPrefix()) to detect URLs without http:// or https:// prefixes.
• enableIpv4Check(true): Detects IPv4 addresses.

// Example: URL detection without prefix
final SensitiveWordBs sensitiveWordBs = SensitiveWordBs.newInstance()
        .enableUrlCheck(true)
        .wordCheckUrl(WordChecks.urlNoPrefix())
        .init();

List<String> wordList = sensitiveWordBs.findAll("baidu.com");

To filter out phone numbers, QQ numbers, or other advertising sequences, enable enableNumCheck(true). You can control the minimum length of the number sequence to avoid false positives using numCheckLen(int length). The default length is 8.

// Example: Detect numbers with a specific length requirement
List<String> wordList2 = SensitiveWordBs.newInstance()
        .enableNumCheck(true)
        .numCheckLen(9)
        .init()
        .findAll(text);

In sensitive-word, a single sensitive word can be associated with multiple tags. Use these predefined tag identifiers to categorize sensitive words in your configuration or data files:

• privacy: Personal privacy related content.
• sex: Pornographic or sexual content.
• violence: Violent content.
• political: Political content.
• cult: Cult-related content.
• ad: Advertising or spam content.

The wordFailFast setting controls how the engine handles overlapping sensitive words.

• wordFailFast=true (Default): Optimized for performance. It returns the first match it finds. This may cause shorter words to be matched instead of longer, more specific ones (e.g., matching "我的" instead of "我的世界").
• wordFailFast=false (failOver mode): Prioritizes finding the longest possible match, which is often more intuitive for content moderation.

// failOver mode (longest match)
SensitiveWordBs bs = SensitiveWordBs.newInstance()
        .wordFailFast(false)
        .wordDeny(new IWordDeny() {
            @Override
            public List<String> deny() {
                return Arrays.asList("我的世界", "我的");
            }
        }).init();