PolyFile Documentation

PolyFile includes an interactive debugger (modeled after GDB) to step through libmagic DSL specifications and parsers.

To enter the debugger, run PolyFile with the --debugger or -db flag:

$ polyfile -db input_file

Available Debugger Commands:

• breakpoint: list or add breakpoints
• continue (or run): continue execution until the next breakpoint
• debug_and_continue: continue while debugging in PDB
• debug_and_rerun: re-run the last test and debug in PDB
• debug_and_step: step into the next magic test and debug in PDB
• delete: delete a breakpoint
• help: print help
• next: continue execution until the next test that matches
• print: print the computed absolute offset of a libmagic DSL offset
• profile: print current profiling results (enable via set profile True)
• step: step through a single magic test
• test: test the following libmagic DSL test at the current position
• where (or backtrace, info stack): print the context of the current magic test
• set / show: modify or print the debugger environment
• quit: exit the debugger

If you want to avoid the libmagic DSL, you can implement a custom matcher by subclassing MagicTest. You must implement the test method, which performs the actual byte sequence validation.

To make the matcher active, register it with MagicMatcher.DEFAULT_INSTANCE.add().

from typing import Optional
from polyfile.magic import AbsoluteOffset, FailedTest, MagicMatcher, MagicTest, MatchedTest, TestResult, TestType

class ExampleMatcher(MagicTest):
    def __init__(self):
        super().__init__(
            offset=AbsoluteOffset(0),
            mime="application-x/example-mime",
            extensions=("example",),
            message="A message that will be printed when this test matches an input"
        )

    def subtest_type(self) -> TestType:
        return TestType.BINARY

    def test(self, data: bytes, absolute_offset: int, parent_match: Optional[TestResult]) -> TestResult:
        if data.startswith(b"example"):
            return MatchedTest(self, value=data, offset=0, length=len(data))
        else:
            return FailedTest(self, offset=0, message="This is not an example file!")

# Register the matcher so it always runs:
MagicMatcher.DEFAULT_INSTANCE.add(ExampleMatcher())

Parsers produce a syntax tree of Submatch objects for classified MIME types. To implement a parser, subclass polyfile.Parser and implement the parse method.

Requirements:

• The parse method must accept a FileStream and a Match object.
• It must yield Submatch objects.
• Each Submatch must be yielded after all of its ancestors have been yielded.
• If the parser determines the classification was incorrect, it should raise polyfile.InvalidMatch.

from polyfile import InvalidMatch, Submatch

def parse(self, stream, match):
    header_content = stream.read(len(b"example"))
    if header_content == b"example":
        yield Submatch(
            name="Example Header",
            match_obj=header_content,
            relative_offset=0,
            length=len(header),
            parent=match
        )
        remaining_content = stream.read()
        content_node = Submatch(
            name="Content",
            match_obj=remaining_content,
            relative_offset=len(header),
            length=len(remaining_content),
            parent=match
        )
        yield content_node
        # ... further submatches ...
    else:
        raise InvalidMatch("The file does not start with b\"example\"!")

PolyFile provides a pure-Python implementation of libmagic via the MagicMatcher class. You can use it to identify MIME types in your own Python applications.

Using the default instance

MagicMatcher.DEFAULT_INSTANCE automatically loads all standard file definitions.

Using custom definitions

To load specific or custom file definitions, use MagicMatcher.parse(*list_of_paths_to_definitions).

from polyfile.magic import MagicMatcher

# Using the default instance
with open("file_to_test", "rb") as f:
    for match in MagicMatcher.DEFAULT_INSTANCE.match(f.read()):
        for mimetype in match.mimetypes:
            print(f"Matched MIME: {mimetype}")
        print(f"Match string: {match!s}")

# Loading custom definitions
list_of_paths_to_definitions = ["def1", "def2"]
matcher = MagicMatcher.parse(*list_of_paths_to_definitions)
with open("file_to_test", "rb") as f:
    for match in matcher.match(f.read()):
        # process match...
        pass

PolyFile outputs its mapping to STDOUT using a JSON schema that extends the SBuD format.

Key differences and extensions from SBuD include:

• struc is a list: While SBuD uses a single object, PolyFile uses a list in the struc field to support labeling multiple filetypes in the case of a polyglot file.
• img_data: An optional field within sub-elements containing base64 encoded image data (not present in SBuD).

Top-level fields include:

• MD5, SHA1, SHA256: Hex strings of the input file hashes.
• b64contents: Base64 encoded contents of the entire input file.
• fileName: The input filename, or 'STDIN' if read from standard input.
• length: Integer number of bytes in the file.
• struc: A list of detected filetype objects.

{
  "MD5": "MD5 hex string of the input file", 
  "SHA1": "SHA1 hex string for the input file", 
  "SHA256": "SHA256 hex string for the input file", 
  "b64contents": "base64 encoded contents of the input file", 
  "fileName": "The input filename, or 'STDIN' if the file was read from STDIN",
  "length": 1337,
  "struc": [
    {
      "name":   "ADOBE_PDF",
      "offset": 0,
      "subEls": [
        {
          "offset": 0,
          "relative_offset": 0,
          "name": "header",
          "type": "magic",
          "size": 9,
          "value": "%PDF1.3\n",
          "img_data": "Optional base64 encoded image",
          "subEls": []
        }
      ] 
    }
  ]
}