Magika AI File Type Detection

Magika provides consistent prediction data across the CLI and language bindings (Python, Rust, JavaScript). When scanning files, the output contains metadata about the file path, the scan status, and the prediction details.

Core Fields

• path: The file path being predicted (useful when scanning multiple files).
• result.status: Indicates if the scan was successful. A value of ok means the value field is present. If the status is not ok, the value field will be absent.
• value.score: A float representing the model's confidence in the prediction.
• value.dl: The raw prediction from the deep learning model.
• value.output: The final prediction from "Magika the tool." This is the recommended field for most applications as it synthesizes the deep learning model's prediction, the confidence score, and the selected prediction mode.

Metadata Fields

Both the dl and output blocks contain the following metadata for the predicted content type:

• label: A machine-readable content type label (e.g., javascript). Note: For most applications, you should use output.label.
• description: A human-readable description of the type.
• mime_type: The corresponding MIME type.
• group: A high-level category (e.g., code, document, media).
• is_text: A boolean indicating if the content is textual.
• extensions: A list of common file extensions associated with the type.

Special Cases

• Empty Files/No Model Use: If the deep learning model is not used (e.g., for empty files), dl.label will be undefined, and the output block will contain a generic content type like txt or unknown.

$ magika tests_data/basic/javascript/code.js --json
[
  {
    "path": "tests_data/basic/javascript/code.js",
    "result": {
      "status": "ok",
      "value": {
        "dl": {
          "description": "JavaScript source",
          "extensions": [
            "js",
            "mjs",
            "cjs"
          ],
          "group": "code",
          "is_text": true,
          "label": "javascript",
          "mime_type": "application/javascript"
        },
        "output": {
          "description": "JavaScript source",
          "extensions": [
            "js",
            "mjs",
            "cjs"
          ],
          "group": "code",
          "is_text": true,
          "label": "javascript",
          "mime_type": "application/javascript"
        },
        "score": 0.9710000157356262
      }
    }
  }
]

Magika models are trained to detect specific sets of content types. As models evolve, newer versions typically support a superset of the content types found in previous versions.

Key Distinctions in Content Type Lists

When reviewing model documentation, you will encounter three different ways content types are categorized:

1. Model Output Space: The specific set of content types the model is trained to detect.
2. Magika Tool Output Space: A superset of the model's output space that includes additional labels such as empty and directory.
3. Knowledge Base (KB): A comprehensive list of all content types tracked internally for research. Do not use the KB as a reference for supported types, as it is a superset of what any single model can actually detect.

How to verify supported types

To determine exactly which content types a specific model supports, always refer to that specific model's README file rather than the general knowledge base. For example, the standard_v3_3 model supports over 200 content types.

Model Evolution

Improvements, tradeoffs, and changes between model versions are documented in the models/CHANGELOG.md. While most clients and bindings integrate the latest available model, you should verify the version used in your specific binding or CLI implementation.

Magika uses per-content-type confidence thresholds rather than a single global threshold. This accounts for the fact that the model's confidence varies naturally by file type (e.g., a 99% score might be typical for PDFs, while 80% might be reliable for JavaScript).

To balance precision (accuracy) and recall (the number of files identified), you can choose between three prediction modes:

1. high-confidence: Prioritizes precision. It only returns predictions that meet high, type-specific confidence thresholds. This results in fewer total identifications but higher accuracy.
2. medium-confidence: A balanced mode between precision and recall.
3. best-guess: Prioritizes recall. It returns the model's top prediction regardless of the confidence score, providing the highest number of identifications but with a higher risk of misidentification.

These modes can be selected via command-line flags or as options in the language bindings.

All identification methods return a MagikaResult object.

MagikaResult Structure

• ok (bool): True if identification was successful.
• status (Status): Details about errors if ok is False.
• prediction (MagikaPrediction): The core prediction object (available only if ok is True).
• Shortcuts (available only if ok is True):
  • dl: Shortcut to prediction.dl.
  • output: Shortcut to prediction.output (the recommended result to use).
  • score: Shortcut to prediction.score.

MagikaPrediction Structure

• dl (ContentTypeInfo): The raw prediction from the deep learning model.
• output (ContentTypeInfo): The final prediction from Magika, which considers the model's prediction, confidence, and prediction mode. This is the result most users should rely on.
• score (float): The model's confidence score (0.0 to 1.0).
• overwrite_reason (OverwriteReason): Why the deep learning prediction was overwritten.

ContentTypeInfo Structure

Contains metadata about a predicted type:

• label: A ContentTypeLabel (e.g., "python").
• mime_type: The MIME type string (e.g., "text/x-python").
• group: The category (e.g., "code").
• description: A human-readable description.
• extensions: A list of associated file extensions.
• is_text: Boolean indicating if the content is text-based.

Magika uses a compact deep learning model optimized for CPU inference to identify file types. The process is highly efficient because it does not read entire files into memory; instead, it inspects only the first few hundred bytes (typically up to 2KB). This allows for fast, constant-time inference regardless of the total file size.

The identification workflow:

1. Byte Extraction: Magika reads initial chunks of the file or byte stream.
2. Feature Extraction: Features are extracted from these bytes for model processing.
3. Prediction: The deep learning model predicts the content type.
4. Confidence Evaluation: Magika checks the model's confidence score against a predefined threshold.
   • High Confidence: Magika accepts the specific predicted type.
   • Low Confidence: Magika returns a generic label (e.g., txt for text or unknown for binary).

Note on Labels: Magika distinguishes between the raw label from the deep learning model and the final label produced by "Magika the tool." The tool's label is the one typically exposed to users, which may be more generic than the model's raw prediction if confidence is low.