Extended OpenAI Conversation

repository·develop·Indexed 23 days ago

https://github.com/jekalmin/extended_openai_conversation

A custom Home Assistant component that extends the standard OpenAI Conversation integration. It enables LLMs to perform complex tasks via OpenAI function calling, including calling Home Assistant services, creating automations, fetching external API data, retrieving entity history, and querying images.

Tokens
65.7K
Snippets
149
Records
218
Agent score
79%

What's inside extended_openai_conversation

  1. Introduction to Extended OpenAI Conversation

    develop

    Extended OpenAI Conversation is a Home Assistant custom component designed to enhance the official OpenAI Conversation integration. It enables the AI to interact with your smart home through advanced capabilities like service calls, automation creation, and data retrieval.

    Key Capabilities

    • Service Calls: Execute any Home Assistant service using natural language.
    • Automation Creation: Dynamically generate automations using AI-generated YAML.
    • External Data: Fetch information from web pages and external APIs.
    • State History: Analyze historical entity states and trends.
    • Skills System: Use reusable modules to extend AI functionality.
    • Diverse Function Types: Supports native, template, script, REST, scrape, composite, and SQLite functions.
  2. How the edit_file function works and its safety features

    develop

    The edit_file function follows a strict read-modify-write lifecycle to ensure safety:

    1. File Read: The entire file is read into memory to verify existence.
    2. Exact Match Search: It searches for an exact string match of old_text (including whitespace, case sensitivity, and special characters).
    3. Occurrence Validation:
      • If old_text is found 0 times $\rightarrow$ Error: "Text not found in file".
      • If old_text is found 1 time $\rightarrow$ Proceed.
      • If old_text is found 2+ times $\rightarrow$ Error: "Text appears N times in file".
    4. Atomic Replacement: The single occurrence is replaced with new_text and the file is written back. If any step fails, the file remains unchanged.

    Return Values

    On Success:

    {
      "success": true,
      "path": "/config/extended_openai_conversation/config.yaml",
      "replacements": 1
    }

    On Error:

    {
      "error": "Text not found in file: old text here..."
    }

    OR

    {
      "error": "Text appears 3 times in file. Please provide more specific text to ensure single replacement."
    }
  3. Configure working directories for write_file

    develop

    The default working directory for write_file is config/extended_openai_conversation/.

    • Relative Path: Writing to notes.txt targets /config/extended_openai_conversation/notes.txt.
    • Subdirectory: Writing to data/config.json targets /config/extended_openai_conversation/data/config.json. Note: The parent directory must already exist.
    • Absolute Path: Writing to /config/scripts/generated.py targets that specific absolute location (provided it is in the allowed directories).
    # Relative Path
    function:
      type: write_file
      path: "notes.txt"
      content: "{{ note_text }}"
    
    # Subdirectory (Parent must exist)
    function:
      type: write_file
      path: "data/config.json"
      content: "{{ json_data }}"
    
    # Absolute Path
    function:
      type: write_file
      path: "/config/scripts/generated.py"
      content: "{{ python_code }}"
  4. How Extended OpenAI Conversation works

    develop

    The component utilizes OpenAI's function calling feature to bridge the gap between natural language and Home Assistant actions.

    To enable control over your devices, you must expose your entities through the Home Assistant Voice Assistants interface. The AI uses these exposed entities to understand your home's capabilities and automatically generate the appropriate service calls (e.g., identifying that 'Turn on the living room lights' requires a light.turn_on service call).

  5. Manage AI Skills

    develop

    Skills are reusable AI capabilities loaded from <config directory>/extended_openai_conversation/skills/.

    To use skills:

    1. Download a skill using the download_skill service.
    2. Enable the skill via Settings > Voice Assistants > Edit Assistant > Options.
    service: extended_openai_conversation.download_skill
    data:
      skill_name: crypto
  6. Return service results to AI using response_variable

    develop

    To allow the AI to see and process the data returned by a Home Assistant service (e.g., calendar events or weather forecasts), you must set response_variable: _function_result within the service call in your sequence. The content of this variable is then passed back to the AI as the function result.

    sequence:
      - service: weather.get_forecasts
        data:
          type: daily
        target:
          entity_id: weather.home
        response_variable: _function_result
  7. Configure exposed entity attributes using customize_glob_exposed_attributes

    develop

    You can control which attributes of a Home Assistant entity are exposed to the LLM by defining a customize_glob_exposed_attributes dictionary. This works similarly to Home Assistant's customize_glob and uses regular expressions (regex) to match entity IDs.

    Logic for attribute inclusion:

    • Boolean true: The actual value of the attribute from the entity is included.
    • Boolean false: The attribute is excluded.
    • Non-boolean value: The specific value you provide in the dictionary is included instead of the actual attribute value from the entity.

    This allows you to filter sensitive data or provide specific context (like a list of allowed media sources) to the model.

    {%- set customize_glob_exposed_attributes = {
      ".*": {
        "friendly_name": true,
      },
      "timer\..*": {
        "duration": true,
      },
      "sun.sun": {
        "next_dawn": true,
        "next_midnight": true,
      },
      "media_player.YOUR_WEBOS_TV": {
        "source_list": ["Netflix","YouTube","wavve"],
        "source": true,
      },
    }
    %}
  8. Jinja2 Template Features for Template Functions

    develop

    Template functions support standard Home Assistant Jinja2 features. Common patterns include:

    • Accessing Entity States: Using states('entity_id'), states.domain.object.state, or state_attr('entity_id', 'attribute').
    • Filtering and Selecting: Using filters like selectattr to find entities in specific states.
    • Date and Time: Using now() and related time functions.
    • Math and Logic: Performing calculations and conditional logic (if/else).
    {# Access Entity States #}
    {{ states('sensor.temperature') }}
    {{ states.sensor.temperature.state }}
    {{ state_attr('sensor.temperature', 'unit_of_measurement') }}
    
    {# Filter and Select #}
    {% set on_lights = states.light | selectattr('state', 'eq', 'on') | list %}
    {{ on_lights | length }} lights are on
    
    {# Date and Time #}
    {{ now() }}
    {{ now().strftime('%Y-%m-%d %H:%M:%S') }}
    {{ as_timestamp(now()) }}
    
    {# Math and Logic #}
    {% set total = (value1 | float) + (value2 | float) %}
    {% if total > 100 %}High{% else %}Normal{% endif %}
  9. Create composite functions using multiple steps

    develop

    A composite function type allows you to execute a sequence of operations (e.g., multiple SQL queries and a template step) to produce a single result.

    In a sequence:

    1. Use type: sqlite to run queries. Use response_variable to store the result of a query for use in subsequent steps.
    2. Use type: template with a value_template to process the variables collected from previous steps into a final response format.
    - spec:
        name: get_states_between
        description: >
          Use this function to get non-numeric states between two dates.
        parameters:
          type: object
          properties:
            entity_id:
              type: string
              description: The target entity
            # ... other properties ...
          required:
            - entity_id
            - start_datetime
            - end_datetime
            - order
            - page
            - limit
      function:
        type: composite
        sequence:
          - type: sqlite
            query: >-
              # ... SQL query using {{parameters}} ...
            response_variable: data
          - type: sqlite
            single: true
            query: >-
              # ... SQL query to get count ...
            response_variable: total
          - type: template
            value_template: '{"data": {{data}}, "total": {{total.count}}}'
  10. How functions are structured in Extended OpenAI Conversation

    develop

    Functions are the primary way to extend your AI assistant's capabilities. Every function is defined using a two-part structure consisting of a spec (the AI's interface) and a function (the execution logic).

    1. spec: Follows the OpenAI function schema format. It tells the AI what the function is and how to call it. It includes:

      • name: The identifier the AI uses to call the function.
      • description: A detailed explanation of the function's purpose (crucial for the AI to decide when to use it).
      • parameters: A JSON schema defining the expected input properties.
    2. function: Defines how the action is actually performed. It includes:

      • type: The specific execution engine to use (e.g., native, rest, bash).
      • Type-specific configuration settings.
    - spec:
        name: function_name
        description: What this function does
        parameters:
          type: object
          properties:
            param_name:
              type: string
              description: Parameter description
      function:
        type: function_type
        # Type-specific configuration
  11. Security constraints for read_file

    develop

    The read_file function implements several security layers:

    1. Path Restriction: Files must be within allowed directories. The default allowed directory is /config/extended_openai_conversation/. Use allow_dir to grant access to other specific paths.
    2. Path Traversal Protection: Paths are resolved to absolute paths and validated. Attempts to use ../ to escape allowed directories (e.g., path: "../../../etc/passwd") are blocked.
    3. Size Limits: Files over 1 MB are automatically rejected to prevent memory exhaustion.
    4. Encoding: Files are read using UTF-8 encoding. Binary files may not be read correctly and should be handled via bash functions (e.g., using xxd).
  12. Add a delay to any function using the reserved delay parameter

    develop

    You can introduce a delay to any function call by adding a reserved delay parameter to the function's specification. This parameter allows you to specify a wait time before the function is executed. The delay parameter is an object containing hours, minutes, and seconds properties.

    To implement this, include the delay object within the properties section of your function's parameters spec.

    - spec:
        name: ...
        description: ...
        parameters:
          type: object
          properties:
            delay: # Add delay parameter to any function spec.
              type: object
              description: Time to wait before execution
              properties:
                hours:
                  type: integer
                  minimum: 0
                minutes:
                  type: integer
                  minimum: 0
                seconds:
                  type: integer
                  minimum: 0
            ...
      function:
        type: ...