DOCSEARCHbyAlgolia
Submit Docs

deid

repository·master·Indexed 41 days ago

https://github.com/pydicom/deid

A Python library providing best-effort anonymization tools for medical images and text. It identifies and cleans sensitive information in medical imaging data, such as DICOM headers, using DeidRecipe configuration files to replace, add, or remove identifiers.

Tokens
22.7K
Snippets
78
Records
109
Agent score
52%

What's inside deid

  1. Overview of De-identification (deid) capabilities

    master

    The deid module is a Python-based tool designed for the anonymization of medical images (DICOM and NIfTI). It focuses on 'cleaning' image headers and pixel data to facilitate de-identification.

    Key Capabilities:

    • Header Anonymization: Replaces, blanks, removes, or applies custom functions to header data based on specific logic.
    • Pixel Data Filtering: Can black out pixel coordinates if they are available, or pass images through a filter for quarantine based on header logic.
    • Customizable Logic: Supports default behaviors using blacklists, whitelists, or graylists, as well as user-defined custom logic.
    • Deployment Options: Provides Python functions for developers, as well as executables and Docker containers for end-users.

    Limitations:

    • It is not a workflow manager.
    • It does not implement custom API calls to retrieve identifiers from external databases.
    • It does not guarantee IRB-validated outputs.

  2. Understand the deid documentation site structure

    master

    The documentation site is built with Jekyll. The following directory structure defines how the site is organized:

    • _config.yml: The primary configuration file. Variables defined here are accessible in templates via {{ site.var }}.
    • _layouts/: Contains base HTML templates for pages.
    • _includes/: Contains HTML snippets used within layouts.
    • pages/: Contains generic pages that are not part of the main documentation (e.g., changelog).
    • _docs/: The collection of folders that are rendered into the documentation sidebar and pages.
    • assets/: Contains all static assets (images, CSS, etc.).
    • _data/: Contains data files (in .yml or .csv format) used to render content into the site.

  3. Understand the Deid DICOM Pipeline

    master

    A complete de-identification process in deid follows a specific sequence of stages to clean header and image data, and filter files based on metadata. The typical pipeline consists of:

    1. Loading Data: Reading the source DICOM files.
    2. Configuration: Defining how the software should handle various DICOM fields via a custom recipe.
    3. Get Identifiers: Extracting specific data (identifiers) from the headers that are intended to be modified.
    4. Clean Pixels: Using header information to flag or process pixel data (e.g., scraping or masking) before finalization.
    5. Put Identifiers: Writing the cleaned/modified header information back into the DICOM images.

    Note that deid algorithms are based on parsing header metadata and are not considered highly sophisticated.

  4. Understand DicomParser recipe structure

    master

    A deidentification recipe is a configuration that defines how to derive values and how to modify the DICOM header. It typically contains three main sections:

    1. values: A list of actions used to derive specific lists of values from the existing data (e.g., extracting all names that start with 'Operator').
    2. fields: A list of actions used to extract specific DICOM fields for later use in the recipe.
    3. header: The sequence of actions to perform on the DICOM file, such as ADD (adding a new field), REPLACE (replacing a value with a variable or derived value), or REMOVE (deleting a field).

    Common action types include SPLIT, FIELD, ADD, REPLACE, and REMOVE.

  5. Understand the deid header cleaning application flow

    master

    De-identifying DICOM headers follows a four-step lifecycle:

    1. Define rules: Create a set of rules for updating values (this is what 'deid recipes' handle).
    2. Get: Retrieve current fields from the DICOM data (useful if replacements depend on existing values).
    3. Update: Modify the retrieved fields based on your rules.
    4. Put: Write the updated identifiers back into the data to complete the de-identification.

    If you are looking for the command-line interface to execute these steps, refer to the client documentation.

  6. Configure deid using a specification file

    master

    Deid configuration files use a custom syntax to define filters and header transformations.

    Structure:

    • FORMAT: Specifies the image format (e.g., dicom).
    • %filter [name]: Defines a named filter group. Filters are processed from most specific to most general. If a filter matches, it becomes the final decision for that file.
      • LABEL [text]: A label for the filter.
      • [field] [operator] [value]: Criteria for matching (e.g., contains PatientSex M).
      • +: Indicates an AND relationship between criteria groups.
      • coordinates [x,y,w,h]: Specifies pixel coordinates for the filter.
    • %header: Defines rules for replacing or adding information in the DICOM header.
      • ADD [field] [value]: Adds a new field to the header.
      • REPLACE [field] var:[variable_name]: Replaces a field value with a variable.

    Example Configuration:

    FORMAT dicom

%filter dangerouscookie
LABEL Criteria for Dangerous Cookie
contains PatientSex M
  + notequals OperatorsName bold bread
  coordinates 0,0,512,110

%header
ADD PatientIdentityRemoved YES
REPLACE PatientID var:id

  7. Create a DeidRecipe for DICOM header manipulation

    master

    A DeidRecipe is defined using a configuration file (e.g., deid.dicom) that specifies the format and a series of actions to perform on the DICOM header.

    Common actions include:

    • REPLACE <Field> <Value>: Replaces the value of an existing field. If the field does not exist, no action is taken.
    • ADD <Field> <Value>: Adds a new field with the specified value.
    • REMOVE <Field>: Removes a field.

    To use a custom Python function for generating values, use the syntax func:<function_name> in the recipe file. The function name must match a key in the ids dictionary passed to the replacement engine.

    Example deid.dicom content:

    FORMAT dicom

%header

REPLACE StudyInstanceUID func:generate_uid
REPLACE SeriesInstanceUID func:generate_uid
ADD FrameOfReferenceUID func:generate_uid
    from deid.config import DeidRecipe

# Load the recipe from the configuration file
recipe = DeidRecipe('deid.dicom')

# Inspect the recipe
print(recipe.get_format())  # e.g., 'dicom'
print(recipe.get_actions())  # Returns list of action dicts
print(recipe.get_actions(field='FrameOfReferenceUID'))

  8. Configure recipe sections and actions

    master

    Recipes use sections to target parts of the data (e.g., %header for DICOM headers) and actions to define what to do with fields within those sections.

    Supported Actions

    • ADD: Adds a new field. Can take a literal value or a variable (var:FieldName).
    • BLANK: Clears the field value (this is the default behavior for many sections).
    • KEEP: Prevents the field from being modified, removed, or blanked.
    • REPLACE: Replaces the field with a string or a variable (var:FieldName).
    • JITTER: Deviates the value by a specified amount (e.g., JITTER Date 31).
    • REMOVE: Completely removes the field from the dataset.

    Action Precedence

    If multiple conflicting commands are applied to the same field, the following order of preference applies (from most conservative to least): KEEP > ADD > REPLACE > JITTER > REMOVE > BLANK

  9. Understand deid recipe sections

    master

    Deid recipes are organized into sections, each identified by a % prefix. These sections define different types of processing logic:

    SectionDescription
    %filterA named set of filter criteria used when running the DicomCleaner. Example: %filter filterName
    %headerActions to be taken to update, or otherwise change, an image header. Example: %header
    %labelsExtra metadata (key-value pairs) to add to a recipe. Example: Maintainer @vsoch

  10. Understand the logic for detecting burned-in pixels

    master

    The pixel cleaning process identifies images that likely have Protected Health Information (PHI) burned into the pixel data. This is achieved using a rule-based approach inspired by the CTP (Clinical Trial Processor) filters.

    An image is considered 'safe' to continue processing (not flagged for quarantine) if it passes a series of header checks. Specifically, an image is processed normally if:

    • ImageType (0008,0008) does not contain "SAVE"
    • DateofSecondaryCapture (0018,1012) is empty/not present
    • SeriesDescription (0008,103e) does not contain "SAVE"
    • SecondaryDeviceCaptureManufacturer (0018,1016) is not present
    • SecondaryDeviceCaptureManufacturerModelName (0018,1018) is not present
    • SecondaryDeviceCaptureDeviceSoftwareVersion (0018,1019) is not present
    • BurnedInAnnotation (0028,0301) is not "YES"

    If any of these conditions fail, the image is flagged for having burned-in PHI and is quarantined.

  11. Configure coordinate rules in a deid recipe

    master

    Deid uses recipes to define how to identify and handle pixel regions. You can use two coordinate standards:

    • CTP standard: Uses ctpcoordinate or ctpkeepcoordinate.
    • Deid standard: Uses coordinate or keepcoordinate with the format xmin, ymin, xmax, ymax.

    Operations:

    • coordinates: Sets the mask value to 0 (indicates the region should be cleaned/scrubbed).
    • keepcoordinates: Sets the mask value to 1 (indicates the region should be kept).

    Example Rule (Scrubbing a box based on metadata):

    LABEL LightSpeed Dose Report
  contains ManufacturerModelName LightSpeed VCT
  + contains Modality CT
  + contains ImageType SCREEN SAVE || contains SeriesDescription Dose
  coordinates 0,0,512,121

    Example Rule (Extracting coordinates from a DICOM field): You can use the from: keyword to extract coordinates directly from a DICOM attribute (e.g., SequenceOfUltrasoundRegions).

    LABEL Clean Ultrasound
    present SequenceOfUltrasoundRegions
    keepcoordinates from:SequenceOfUltrasoundRegions

  12. Understand default de-identification behavior

    master

    By default, the application follows a conservative de-identification process:

    1. All fields are returned for inspection.
    2. You can choose to replace none or all of these fields with your preferred identifiers.
    3. The data is rewritten with your specified changes, and all other fields are blanked.
    4. A header field is added to indicate that the data has been anonymized.

    To deviate from this, you can define specific actions for headers or perform custom functions between the get, update, and put stages.