Octopii PII Scanner Documentation

repository·master·Indexed 20 days ago

Octopii is a PII scanner designed to automate the discovery and extraction of sensitive data, such as Government IDs, addresses, and contact information, from images, PDFs, and documents. It supports scanning local files, S3 URLs, Apache open directory listings, and individual URLs. The tool utilizes Tesseract OCR, OpenCV for face detection and image cleaning, and Spacy/NLTK for NLP-based extraction, outputting results in JSON format.

Tokens: 867
Snippets: 3
Records: 4
Agent score: 23%

What's inside Octopii

How Octopii processes images and documents
master
Octopii follows a multi-step pipeline to detect and extract PII:
1. Input & Importing:
  Images (jpg, png): Detected via PIL and opened with OpenCV.
  PDFs: Converted into a list of images for OCR scanning.
  Text files (doc, txt, etc.): Read directly as strings without OCR.
2. Face Detection: Uses a Haar cascade model via OpenCV to detect faces. This happens before image cleaning to preserve image data.
3. Image Cleaning: Images are transformed to improve OCR accuracy through: auto-rotation, grayscaling, monochrome conversion, mean thresholding, Gaussian thresholding, and deskewing.
4. OCR & Extraction:
  Tesseract extracts text strings.
  Similarity Matching: Extracted words are compared against definitions.json using Gestalt pattern matching to determine the pii_class.
  Regex & NLP: Regular expressions find emails, phone numbers, and IDs, while Natural Language Processing (NLTK/Spacy) extracts addresses and countries.
https://github.com/redhuntlabs/octopii
Run Octopii to scan for PII
master
Run Octopii by passing a target location to the octopii.py script. The <location to scan> can be a single file or a directory.
Supported scan sources:
- Local filesystem: Files and folders within UNIX-like filesystems (macOS/Linux).
- S3 URLs: Traverses XML from Amazon S3 container URLs.
- Apache open directory listings: Traverses and scans files in open directories.
- Individual URLs: You can provide individual image URLs as arguments.
```
python3 octopii.py <location to scan>
```
https://github.com/redhuntlabs/octopii
Install Octopii and its dependencies
master
To set up Octopii, you must install Python dependencies, the Tesseract OCR engine, and the Spacy language model. Follow these steps:
1. Install Python requirements:
  pip install -r requirements.txt
2. Install Tesseract OCR (system-level):
  Ubuntu: sudo apt install tesseract-ocr -y
  Arch Linux: sudo pacman -Syu tesseract
3. Download the Spacy English language model:
  python -m spacy download en_core_web_sm
```
pip install -r requirements.txt
sudo apt install tesseract-ocr -y
python -m spacy download en_core_web_sm
```
https://github.com/redhuntlabs/octopii
Understand the Octopii PII output format
master
Octopii scans files and outputs a JSON object for each detected PII instance. Results are also appended in real-time to a file named output.txt.
Each output object contains the following keys:
- file_path: The path/URL to the file containing PII.
- pii_class: The detected type of PII (e.g., Nebraska Driver's License).
- country_of_origin: The country where the PII originates.
- faces: The number of faces detected in the image.
- identifiers: A list of unique identifiers, codes, or numbers.
- emails: A list of extracted email addresses.
- phone_numbers: A list of extracted phone numbers.
- addresses: A list of extracted geolocation data or addresses.
```
{
    "file_path": "dummy-pii/dummy-drivers-license-nebraska-us.jpg",
    "pii_class": "Nebraska Driver's License",
    "country_of_origin": "United States",
    "faces": 1,
    "identifiers": [],
    "emails": [],
    "phone_numbers": [
        "4000002170"
    ],
    "addresses": [
        "Nebraska"
    ]
}
```
https://github.com/redhuntlabs/octopii