Sysdiagnose Analysis Framework (SAF)

The knowledge_app_usage module extracts application usage patterns from the knowledgeC.db database on iOS and macOS platforms. It identifies which applications were in use, their duration, and associated metadata.

Supported Platforms and Versions

• Platforms: IOS, MACOS
• Versions: 12, 13, 10.14, 10.15, 10.16, 14

Extracted Data Fields

When running the knowledge_app_usage query, the following information is retrieved:

• START: The timestamp when application usage began.
• END: The timestamp when application usage ended.
• BUNDLE ID: The unique identifier for the application.
• USAGE IN SECONDS: Total duration of usage in seconds.
• USAGE IN MINUTES: Total duration of usage in minutes.
• DEVICE ID (HARDWARE UUID): The hardware UUID of the device.
• NAME: Metadata name associated with the event.
• VALUE: Metadata value associated with the event.
• DAY OF WEEK: The human-readable day (e.g., Monday, Tuesday).
• GMT OFFSET: The GMT offset in hours.
• ENTRY CREATION: The timestamp when the database entry was created.
• UUID: The unique identifier for the object.
• ZMETADATAHASH: The metadata hash from ZSTRUCTUREDMETADATA.
• ZOBJECT TABLE ID: The primary key (Z_PK) from the ZOBJECT table.

SELECT
   DATETIME(ZOBJECT.ZSTARTDATE+978307200,'UNIXEPOCH') AS "START", 
   DATETIME(ZOBJECT.ZENDDATE+978307200,'UNIXEPOCH') AS "END",
   ZOBJECT.ZVALUESTRING AS "BUNDLE ID", 
   (ZOBJECT.ZENDDATE - ZOBJECT.ZSTARTDATE) AS "USAGE IN SECONDS",
   (ZOBJECT.ZENDDATE - ZOBJECT.ZSTARTDATE)/60.00 AS "USAGE IN MINUTES",  
   ZSOURCE.ZDEVICEID AS "DEVICE ID (HARDWARE UUID)", 
   ZCUSTOMMETADATA.ZNAME AS "NAME",
   ZCUSTOMMETADATA.ZDOUBLEVALUE AS "VALUE",
   CASE ZOBJECT.ZSTARTDAYOFWEEK 
      WHEN "1" THEN "Sunday"
      WHEN "2" THEN "Monday"
      WHEN "3" THEN "Tuesday"
      WHEN "4" THEN "Wednesday"
      WHEN "5" THEN "Thursday"
      WHEN "6" THEN "Friday"
      WHEN "7" THEN "Saturday"
   END "DAY OF WEEK",
   ZOBJECT.ZSECONDSFROMGMT/3600 AS "GMT OFFSET",
   DATETIME(ZOBJECT.ZCREATIONDATE+978307200,'UNIXEPOCH') AS "ENTRY CREATION", 
   ZOBJECT.ZUUID AS "UUID",
   ZSTRUCTUREDMETADATA.ZMETADATAHASH,
   ZOBJECT.Z_PK AS "ZOBJECT TABLE ID" 
FROM ZOBJECT
      LEFT JOIN ZSTRUCTUREDMETADATA ON ZOBJECT.ZSTRUCTUREDMETADATA = ZSTRUCTUREDMETADATA.Z_PK 
      LEFT JOIN ZSOURCE ON ZOBJECT.ZSOURCE = ZSOURCE.Z_PK 
      LEFT JOIN Z_4EVENT ON ZOBJECT.Z_PK = Z_4EVENT.Z_11EVENT
      LEFT JOIN ZCUSTOMMETADATA ON Z_4EVENT.Z_4CUSTOMMETADATA = ZCUSTOMMETADATA.Z_PK
WHERE ZSTREAMNAME = "/app/usage"

The framework automatically generates a ResultSummary for every execution, persisted as <module_name>.summary.json alongside the output file. This summary is managed by ResultSummaryExecutionHandler and includes:

• status: ok, warning, error, or skipped (derived from log handler counts and compatibility).
• start_time: The timestamp when execution began.
• duration: The time taken for execution (not set for skipped status).
• num_events: Total number of entries produced.
• num_errors: Count of logger.error() calls during execution.
• num_warnings: Count of logger.warning() calls during execution.

Use Cases:

1. Trust Signal: Consumers (CLI/API) can check the summary to assess data quality (e.g., error counts) when loading cached results.
2. Staleness Detection: The start_time allows the CLI to detect if cached data is outdated, especially when analysers depend on previously parsed data.

The knowledge_notification_usage module is an APOLLO module designed to extract notification usage patterns from the knowledgeC.db database on iOS. It identifies when notifications were interacted with, the associated Bundle ID, the notification type, and the device hardware UUID.

Supported Platforms and Versions

• Platform: iOS
• Supported Versions: 12, 13, 10.15, 10.16, 14

Extracted Data Fields

When running this query, the following metadata is retrieved:

• START / END: Timestamps of the notification usage activity.
• BUNDLE ID: The identifier for the application that sent the notification.
• NOTIFICATION TYPE: The specific type of notification interaction recorded.
• DEVICE ID (HARDWARE UUID): The hardware UUID of the device.
• ID: The notification identifier.
• DAY OF WEEK: The human-readable day (e.g., Monday, Tuesday).
• GMT OFFSET: The offset from GMT in hours.
• ENTRY CREATION: When the entry was created in the database.
• UUID: The unique identifier for the object.
• ZMETADATAHASH: The metadata hash.
• ZOBJECT TABLE ID: The primary key from the ZOBJECT table.

-- Example of the logic used to extract notification usage
SELECT
   DATETIME(ZOBJECT.ZSTARTDATE+978307200,'UNIXEPOCH') AS "START", 
   DATETIME(ZOBJECT.ZENDDATE+978307200,'UNIXEPOCH') AS "END",
   ZSTRUCTUREDMETADATA.Z_DKNOTIFICATIONUSAGEMETADATAKEY__BUNDLEID AS "BUNDLE ID",
   ZOBJECT.ZVALUESTRING AS "NOTIFICATION TYPE", 
   ZSOURCE.ZDEVICEID AS "DEVICE ID (HARDWARE UUID)",  
   ZSTRUCTUREDMETADATA.Z_DKNOTIFICATIONUSAGEMETADATAKEY__IDENTIFIER AS "ID", 
   -- ... (day of week logic) ...
FROM ZOBJECT
     LEFT JOIN ZSTRUCTUREDMETADATA ON ZOBJECT.ZSTRUCTUREDMETADATA = ZSTRUCTUREDMETADATA.Z_PK 
     LEFT JOIN ZSOURCE ON ZOBJECT.ZSOURCE = ZSOURCE.Z_PK 
WHERE ZSTREAMNAME = "/notification/usage"

To create a parser, inherit from BaseParserInterface. You must implement:

• get_log_files(): Returns a list of absolute paths to the files this parser needs to process. Use self.case_data_subfolder to locate files within the sysdiagnose directory.
• execute(): The core logic. It should return data matching the specified format.

Key Rules for Parsers:

• Guard against missing files: If get_log_files() returns nothing, log a warning and return an empty list/dict. Do not raise exceptions for missing files.
• Return types:
  • format = "json" $\rightarrow$ return dict or list.
  • format = "jsonl" $\rightarrow$ return list[dict] or a Generator[dict] (for large datasets).
  • Custom formats (e.g., "csv") $\rightarrow$ return str.
• Logging: Use logger.warning() or logger.error() for diagnostics. Avoid print(), as it is forbidden and will be caught by linting rules.

import os
from sysdiagnose.utils.base import BaseParserInterface, Event, SysdiagnoseConfig, logger


class MyParser(BaseParserInterface):
    description = "Parsing my_file.txt"
    format = "jsonl"  # "json" for structured data, "jsonl" for event-based timelines

    def __init__(self, config: SysdiagnoseConfig, case: dict):
        super().__init__(__file__, config, case)

    def get_log_files(self) -> list:
        log_files_globs = ["my_file.txt"]
        return [
            os.path.join(self.case_data_subfolder, f)
            for f in log_files_globs
            if os.path.exists(os.path.join(self.case_data_subfolder, f))
        ]

    def execute(self) -> list:
        log_files = self.get_log_files()
        if not log_files:
            logger.warning("No my_file.txt file present")
            return []

        result = []
        for log_file in log_files:
            # Parse the file and build Event objects
            ...
        return result

To create an analyser, inherit from BaseAnalyserInterface. Analysers consume data from parsers.

Implementation Steps:

1. Instantiate Parsers: Use ParserClass(self.config, self.case) to pass through the case metadata.
2. Retrieve Data: Call parser.get_result(). This method automatically handles caching, so it won't re-parse the file if it has already been processed.
3. Execute Logic: Implement the execute() method to process the data and return the result in the format specified by self.format.

Note: Analysers do not implement get_log_files(). Instead, they should guard against empty input data returned by the upstream parsers.

from sysdiagnose.parsers.my_parser import MyParser
from sysdiagnose.utils.base import BaseAnalyserInterface, SysdiagnoseConfig, logger


class MyAnalyser(BaseAnalyserInterface):
    description = "Analyse something useful"
    format = "json"

    def __init__(self, config: SysdiagnoseConfig, case: dict):
        super().__init__(__file__, config, case)

    def execute(self) -> dict:
        parser = MyParser(self.config, self.case)
        data = parser.get_result()  # triggers parsing if not cached
        # Process data...
        return {"analysis": "result"}