BioChatter

BioChatter is a Python framework designed for deploying, testing, and evaluating conversational AI models within the biomedical domain. It provides a generic backend library that connects biomedical applications to conversational AI, enabling the use of diverse models and auxiliary technologies.

Key characteristics include:

• BioCypher Integration: It is part of the BioCypher ecosystem and connects natively to BioCypher knowledge graphs.
• Extensibility: It serves as a bridge between biomedical knowledge and generative AI models.
• Deployment Options: The framework supports different architectural patterns, from agile Python-based prototyping to advanced client-server setups.

The BioChatter benchmark provides living results for evaluating models across various tasks. The results are categorized into three primary views to help developers assess model performance:

1. Scores per model: Evaluates performance based on the specific model used, sorted by median score.
2. Scores per quantisation: Evaluates how different quantization levels affect performance.
3. Scores of all tasks: A comprehensive view showing performance across all individual benchmark tasks.

For detailed methodology on how these benchmarks are conducted, refer to the benchmarking documentation and the developer docs.

Large Language Models (LLMs) like GPT-3.5 or GPT-4 can sometimes struggle with the directionality of relationships in a Knowledge Graph (e.g., confusing (Crime)-[:INVESTIGATED_BY]->(Officer) with (Officer)-[:INVESTIGATED_BY]->(Crime)).

BioChatter compensates for this by using the source and target annotations from the BioCypher YAML schema definition. The prompts.py module uses this metadata to only propose relationship patterns that actually exist in the KG, effectively constraining the LLM to valid edge directions.

BioChatter evaluates LLM responses based on five core validation requirements, tested both with and without Retrieval-Augmented Generation (RAG). These requirements are applied to tasks involving personalized interventional medicine in geroscience:

1. Comprehensiveness
2. Correctness
3. Usefulness
4. Interpretability/Explainability
5. Consideration of Toxicity

BioChatter Light uses a modular architecture based on the Streamlit framework, allowing you to add new tabs or customize existing ones.

To add new functionality:

1. Implement the new UI components in the corresponding module (e.g., components/panels/project.py for custom project-related tabs).
2. Register the new tabs in the configuration file (e.g., components/config.py) to allow them to be toggled via environment variables.

Each tab typically requires implementing Streamlit components like columns, buttons, and text fields.

BioChatter manages LLM interactions through the Conversation class and its specialized child classes located in the llm_connect.py module. These child classes (e.g., GptConversation, AnthropicConversation, GeminiConversation) abstract away the specific API differences between different LLM providers.

A conversation is built by managing a history of messages, including system messages, user messages, and AI responses. The conversation object handles the orchestration of a 'primary model' and an optional 'correcting agent' based on the prompts provided during initialization.

from biochatter.llm_connect import GptConversation

conversation = GptConversation(
    model_name="gpt-3.5-turbo",
    prompts={},
)

Text extraction benchmarks evaluate the ability of LLMs to extract specific structured information from unstructured documents.

Key extraction tasks include:

• Biological/Chemical Entities: Assay, Chemical, Context, Disease, Entity, Gene identifier mapping (NCBI links), Intervention, and Hypothesis.
• Experimental Data: Experiment (yes/no), Significance, and Statistics.
• Multimodal: Image captioning (yes/no).

RAG benchmarks in BioChatter test the ability of LLMs to either generate answers to questions using a RAG agent or to judge the relevance of specific text fragments to a question.

Two testing modes are used:

• Explicit relevance: The model is explicitly asked if a single fragment is relevant to a question.
• Implicit relevance: The model is asked a question without explicit instructions, and performance is evaluated based on whether the model correctly identifies if 'not enough information is given'.