repository·master·Indexed 19 days ago
https://github.com/leapbeyond/scrubadubA Python library designed to remove personally identifiable information (PII) from free text, such as names, emails, and phone numbers. It provides a Scrubber class to manage detectors, including default detectors for credentials, credit cards, and social security numbers, as well as optional extensions like scrubadub_address for British, American, and Canadian addresses. The library includes a comparison module to measure detector performance using precision, recall, and f1-score via classification reports and pandas DataFrames.
Scrubadub supports removing the following types of information:
In scrubadub, a Filth object is a specialized class responsible for two primary tasks:
All specific filth types (like those for emails or phone numbers) inherit from the base scrubadub.filth.Filth class.
When evaluating how accurately scrubadub detects PII (referred to as Filth), use the following metrics:
Filth detected out of all Filth selected by the Detector. Low precision means clean text is being incorrectly flagged as Filth (false positives).Filth that is actually selected by the Detector. Low recall means dirty text is being missed (false negatives).In scrubadub, a Detector is an abstraction responsible for identifying and iterating over Filth (sensitive information) within a piece of text. All detectors inherit from the base scrubadub.detectors.Detector class.
To create custom detection logic based on regular expressions, you can use the RegexDetector or RegionLocalisedRegexDetector base classes, which simplify the process of adding new types of Filth identification.
import scrubadub.detectors
# All detectors inherit from this base class
# base_detector = scrubadub.detectors.Detector()To create a custom detector that responds to specific locales, you must implement a supported_locale(cls, locale) class method.
supported_locale(cls, locale): This method must return True if the provided locale is supported by the detector, and False otherwise. If it returns False, the Scrubber will emit a warning and skip this detector for that document.locale_split(locale): Use this helper method within your detector to split the xx_YY locale string into its language and region components.If supported_locale is not defined, the detector is assumed to be location-independent.
import scrubadub, re
class EmployeeNameFilth(scrubadub.filth.Filth):
type = 'employee_name'
class EmployeeDetector(scrubadub.detectors.Detector):
name = 'employee_detector'
def __init__(self, *args, **kwargs):
super(EmployeeDetector, self).__init__(*args, **kwargs)
# Example: mapping regions to specific names
self.employees = {'DE': ['Walther'], 'US': ['Georgina'] }
# Note: In a real implementation, self.region would be derived from self.locale
self.region = self.locale.split('_')[1]
self.regex = re.compile('|'.join(self.employees[self.region]))
@classmethod
def supported_locale(cls, locale):
# Use locale_split to get language and region
language, region = cls.locale_split(locale)
return region in ['DE', 'US']
def iter_filth(self, text, document_name=None):
for match in self.regex.finditer(text):
yield EmployeeNameFilth(match=match, detector_name=self.name, document_name=document_name, locale=self.locale)In scrubadub, a Filth object is the core abstraction used to identify and handle sensitive information.
Each Filth object has two primary responsibilities:
All specific filth types inherit from the base class scrubadub.filth.base.Filth.
Post Processors are components in Scrubadub that execute after filth (sensitive information) has been detected. They run in a specific order and allow you to manipulate the detected Filth objects. Common use cases include:
You can implement custom logic to handle the Filth objects according to your specific data privacy or processing requirements.
The scrubadub library is composed of four primary components that manage the text cleaning lifecycle:
Filth objects: Represent specific parts of the text identified as containing sensitive information.Detector objects: Responsible for identifying specific types of Filth within the text.PostProcessor objects: Used to alter the identified Filth (e.g., replacing it with a hash, a token, or custom markers).Scrubber: The central manager that coordinates the cleaning process. It maintains the list of Detector and PostProcessor objects, manages Filth objects, and resolves conflicts between different detectors.When building evaluation pipelines, it is important to distinguish between these two detector types:
TaggedFilthDetector: Always returns TaggedEvaluationFilth. This type is specifically designed to serve as the 'ground truth' or 'truth' when calculating classification reports.UserSuppliedFilthDetector: Returns the specific type of filth requested (e.g., EmailFilth or PhoneFilth). These are used to represent the actual detections being evaluated.A PostProcessor is used to process detected Filth objects and modify them after they have been identified. Currently, available post-processors primarily focus on altering the replacement string used when scrubbing data.
Common built-in post-processors include:
FilthReplacer: Alters the replacement string for detected filth.PrefixSuffixReplacer: Adds prefixes or suffixes to the replacement string.FilthRemover: Removes the detected filth entirely from the text.To make a detector available to the Scrubber catalogue, use the @scrubadub.detectors.register_detector decorator.
If you set autoload = True on the detector class, it will be automatically loaded into any Scrubber initialized without a specific detector_list argument. This allows you to use scrubber.remove_detector('name') to manage them dynamically.
For third-party packages, you can register detectors using Python entry points in your setup.cfg:
[options.entry_points]
scrubadub_detectors =
orange = scrubadub_fruit.detectors:OrangeDetectorimport scrubadub, re
@scrubadub.detectors.register_detector
class OrangeDetector(scrubadub.detectors.Detector):
name = 'orange'
autoload = True
# ... implementation ...
scrubber = scrubadub.Sc scrubber()
# OrangeDetector is already loaded due to autoload=True
scrubber.remove_detector('orange')You can specify a locale to ensure detectors (like PhoneDetector or AddressDetector) use the correct regional formats or machine learning models. Locales follow the standard xx_YY format, where xx is a lower-case language code (ISO 639-1) and YY is an upper-case country code (ISO 3166-1 alpha-2).
Examples:
en_US (US English)en_GB (UK English)fr_CA (Canadian French)de_AT (Austrian German)You can set the locale by passing it to the scrubadub.clean() function or when initializing a scrubadub.Scrubber instance.
import scrubadub
# Using the top-level clean function
scrubadub.clean('My US number is 731-938-1630', locale='en_US')
# Using a Scrubber instance
scrubber = scrubadub.Scrubber(locale='de_DE')
scrubber.clean('Meine Telefonnummer ist 05086 63680')