OneAIFW Documentation

The Chinese address recognition system uses a priority-based, bitmap-driven approach to identify and reconstruct complete Chinese addresses from NER (Named Entity Recognition) seeds.

Core Logic:

• Tokenization: Addresses are decomposed into ordered hierarchical levels (from macro to micro).
• Bitmap Representation: A u32 bitset (addr_priorities) tracks which hierarchical levels are covered in an address fragment.
• Expansion Rules: The system expands address fragments using 'Left Expansion' (macro-direction) and 'Right Expansion' (micro-direction) based on 'adjacent level' rules.
• Privacy Threshold: An address is considered a 'private address' (eligible for masking) if its lowest covered level reaches or falls below the privacy threshold (typically L5: House Number or below).
• Language Gating: This logic is only activated for Chinese languages (zh, zh_cn, zh_tw, zh_hk, zh_hans, zh_hant).

To process sensitive text (e.g., translating it via an LLM) while maintaining privacy, you must use the Anonymization and De-anonymization APIs as a pair.

The Workflow:

1. Anonymize: Call /api/mask_text with the original text. It returns a masked_text (containing placeholders) and a maskMeta string.
2. Process: Send the masked_text to your external service (like an LLM).
3. De-anonymize: Call /api/restore_text using the processed text and the exact same maskMeta received in step 1.

Critical Requirements:

• Pairing: Every anonymization call must have a corresponding de-anonymization call to prevent memory leaks.
• maskMeta: This is a Base64 encoded string representing the placeholdersMap. The client should treat it as an opaque string and pass it back to the server exactly as received.

Example (Python):

import requests

base = "http://127.0.0.1:8844"

# 1. Anonymize
r = requests.post(f"{base}/api/mask_text", json={"text": "张三电话13812345678", "language": "zh"})
output = r.json()["output"]
masked_text = output["text"]
mask_meta_b64 = output["maskMeta"]

# 2. Restore
r2 = requests.post(f"{base}/api/restore_text", json={"text": masked_text, "maskMeta": mask_meta_b64})
print("restored:", r2.json()["output"]["text"])

To process text while protecting privacy, use the Mask and Restore interfaces as a pair.

1. Masking: Call /api/mask_text with raw text. The server returns a version of the text where sensitive data is replaced by placeholders (e.g., __PII_EMAIL_ADDRESS_00000001__) and a maskMeta string.
2. Processing: Send the masked text to your external service (e.g., an LLM or translation engine).
3. Restoration: Call /api/restore_text using the masked text from the LLM and the exact same maskMeta received in step 1. This replaces placeholders with the original values.

CRITICAL: You must use the same maskMeta for both calls. maskMeta is a base64-encoded JSON string representing the placeholdersMap. Failure to pair calls correctly may lead to memory leaks in the backend service.

# Python workflow example
# 1. Mask
r = requests.post(f"{base}/api/mask_text", json={"text": "张三电话13812345678", "language": "zh"})
output = r.json()["output"]
masked_text = output["text"]
mask_meta_b64 = output["maskMeta"]

# 2. Restore
r2 = requests.post(f"{base}/api/restore_text", json={"text": masked_text, "maskMeta": mask_meta_b64})
print(r2.json()["output"]["text"])

The py-origin sub-project provides the OneAIFW Python backend and CLI, which uses Presidio and LiteLLM to anonymize sensitive data before LLM calls and restore it afterward.

To set up a local environment:

1. Clone the repository and navigate to py-origin.
2. Create and activate a virtual environment.
3. Install dependencies for both the service and the CLI.
4. Download the required spaCy language models (en_core_web_sm, zh_core_web_sm, and xx_ent_wiki_sm).
5. Configure the global ~/.aifw/aifw.yaml file by copying the asset and setting the api_key_file path to your LLM API key JSON.

# Clone and setup venv
git clone https://github.com/funstory-ai/aifw.git
cd aifw/py-origin
python -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -r services/requirements.txt
pip install -r cli/requirements.txt

# Download spaCy models
python -m spacy download en_core_web_sm
python -m spacy download zh_core_web_sm
python -m spacy download xx_ent_wiki_sm

# Configure aifw.yaml
mkdir -p ~/.aifw
cp assets/aifw.yaml ~/.aifw/aifw.yaml
# Note: Edit ~/.aifw/aifw.yaml to set api_key_file to your LLM API key JSON

The CLI provides several ways to interact with the anonymization service:

1. HTTP-based calls (via the server)

These commands call the running HTTP server. If an API key is required, use --http-api-key.

• python -m aifw call "<text>": Masks PII, calls an LLM, and restores the text.
• python -m aifw mask_restore "<text>": A single pipeline for masking and then restoring a single text.
• python -m aifw mask_restore_batch "<text1>" "<text2>": Batch processing for multiple texts.
• python -m aifw multi_mask_one_restore "<text1>" "<text2>": Masks multiple items individually, then restores them all in one batch.

2. Direct in-process calls (no HTTP server required)

• python -m aifw direct_call "<text>": Performs the mask/LLM/restore cycle directly in-process.

3. Overriding API keys

You can override the LLM API key file for any call using the --api-key-file flag.

# Call via HTTP server
python -m aifw call "My email is test@example.com" --http-api-key 8H234B

# Batch mask/restore via HTTP
python -m aifw mask_restore_batch "text 1" "text 2" --http-api-key 8H234B

# Direct in-process call
python -m aifw direct_call "My email is test@example.com"

# Call with specific API key file
python -m aifw call --api-key-file /path/to/key.json "..."