repository·main·Indexed 26 days ago
https://github.com/giskard-ai/giskard-ossAn open-source Python testing framework for ML models, ranging from tabular data to LLMs. Giskard provides tools for vulnerability scanning, RAG evaluation, and red-teaming agentic AI systems. It includes giskard-agents for orchestrating LLM workflows with asynchronous generators and tools, and giskard-checks for building test scenarios and suites to evaluate Systems Under Test (SUT).
giskard-scan is an agent vulnerability scanner designed for AI agents. It provides capabilities for red teaming, detecting prompt injection vulnerabilities, and generating adversarial scenarios to test the robustness of AI systems.Giskard Agents uses three primary components to orchestrate LLM workflows:
Generator: Represents a conversational text generator (a model with specific parameters) capable of running completions.ChatWorkflow: Defines the configuration for a chat, including templates, parsing logic, and tools.Chat: The result of a pipeline execution, containing generated messages and metadata.Note: The API is entirely asynchronous; all .run() methods return coroutines.
The giskard-llm library distinguishes between input and output types using different base classes:
TypedDict, names ending in Param): Examples include ChatMessageParam, ToolDefParam, FunctionDefParam, and FunctionCallOutputParam. These are lightweight and designed for user construction via dictionary literals (e.g., {"role": "user", "content": "hello"})._BaseModel): Examples include CompletionResponse, Choice, AssistantMessage, ToolCall, ToolCallFunction, EmbeddingResponse, and ResponseResult. These are constructed by provider implementations and support attribute access (e.g., resp.choices[0].message.content) and .model_dump() for serialization.To prevent giskard-core from sending usage analytics, you can use environment variables or a runtime function call.
Important: To ensure no persistent anonymous ID is written to ~/.giskard/id, set the environment variables before importing any giskard packages.
Install giskard-checks to create evaluations (evals) for LLM-based systems. This library supports simple assertions and LLM-as-judge assessments (like Groundedness, Conformity, and LLMJudge) to handle non-deterministic outputs. Use it to catch regressions, validate RAG quality, enforce safety, and evaluate multi-turn agents.
Note: The run() method is asynchronous and should be wrapped with asyncio.run() in scripts.
from openai import OpenAI
from giskard.checks import Scenario, Groundedness
client = OpenAI()
def get_answer(inputs: str) -> str:
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": inputs}],
)
return response.choices[0].message.content
scenario = (
Scenario("test_dynamic_output")
.interact(
inputs="What is the capital of France?",
outputs=get_answer,
)
.check(
Groundedness(
name="answer is grounded",
context="France is a country in Western Europe. Its capital is Paris.",
)
)
)
result = await scenario.run()
result.print_report()Install giskard-scan to perform red-teaming and vulnerability scanning on agentic systems. It automatically generates adversarial test suites covering prompt injection, harmful content, stereotypes, and misinformation based on a plain-language description of your agent.
import asyncio
from giskard.scan import vulnerability_scan
async def main():
await vulnerability_scan(
target=my_agent,
description="A customer support chatbot for an e-commerce platform.",
languages=["en"],
)
asyncio.run(main())You can install giskard-core using uv or pip.
uv add giskard-core
# or
pip install giskard-coreBefore making any changes to the repository, run the make setup-for-agents command to prepare the environment. You must provide the AGENT_NAME and a REASON (describing the issue or task) as arguments. It is recommended to use Makefile targets rather than executing raw Python or pytest commands directly.
make setup-for-agents AGENT_NAME="<name>" REASON="<issue or task>"To ensure compatibility and code quality on every commit, install pre-commit hooks.
# Quick setup (if make setup was run)
make pre-commit-install
# Or manually
uv tool install pre-commit --with pre-commit-uv
pre-commit install
# Run on all files
make pre-commit-runGiskard Agents supports Jinja2 templating for prompts via inline strings or external files.
as_template=True to .chat() and provide variables via .with_inputs().agents.set_prompts_path("path/to/prompts") and load files via .template("filename.j2").# Inline template
chat = await (
generator.chat("Hello {{ name }}, how are you?", as_template=True)
.with_inputs(name="Test Bot")
.run()
)
# External template
agents.set_prompts_path("path/to/the/prompts")
chat = await (
generator.template("hello_template.j2")
.with_inputs(name_of_the_bot="Test Bot")
.run()
)Install the core Giskard library using pip. Requires Python 3.12+.
Note on Telemetry: Libraries built on giskard-core (including giskard-checks) may send optional, aggregated usage analytics to help improve the product. No prompts, model outputs, or scenario text are included.
pip install giskardUse the Scenario class to build a test case using a fluent API. You can provide static values or callables for inputs and outputs. The run() method is asynchronous and must be awaited.
from giskard.checks import Groundedness, Scenario
# Static values
scenario = (
Scenario("test_france_capital")
.interact(
inputs="What is the capital of France?",
outputs="The capital of France is Paris."
)
.check(
Groundedness(
name="answer is grounded",
answer_key="trace.last.outputs",
context="""France is a country in Western Europe. Its capital
and largest city is Paris, known for the Eiffel Tower
and the Louvre Museum."""
)
)
)
result = await scenario.run()
assert result.passed