DOCSEARCHbyAlgolia
Submit Docs

PandasAI

repository·main·Indexed 12 days ago

https://github.com/sinaptik-ai/pandas-ai

A Python library that enables conversational data analysis and natural language interaction with data sources including SQL, CSV, pandas, MongoDB, and noSQL. Version 3.0.0 uses LLMs (GPT 3.5/4, Anthropic, VertexAI) and RAG to perform complex analysis, generate visualizations, and query multiple DataFrames. It includes extensions for various databases (PostgreSQL, MySQL, Snowflake, Oracle), vector stores (ChromaDB, Pinecone, Milvus), and security features like a Docker Sandbox and AdvancedSecurityAgent.

Tokens
59.7K
Snippets
202
Records
248
Agent score
97%

What's inside PandasAI

  1. What is PandasAI?

    main

    PandasAI is a Python library that enables natural language interaction with data. It uses Generative AI models to interpret natural language queries and translate them into Python code or SQL queries to interact with your datasets.

    Key capabilities include:

    • Natural language querying: Ask questions about your data using plain English.
    • Data visualization: Automatically generate graphs and charts.
    • Data cleansing: Address missing values and clean datasets.
    • Feature generation: Enhance data quality through automated feature creation.
    • Data connectors: Connect to various sources including CSV, XLSX, PostgreSQL, MySQL, BigQuery, Databricks, and Snowflake.

  2. What is the Semantic Data Layer in PandasAI 3.0?

    main

    The Semantic Data Layer is an experimental feature introduced in PandasAI 3.0. It allows you to transform raw data into semantic-enhanced, clean dataframes that can be queried via conversational AI dashboards.

    It serves three primary functions:

    1. Data configuration: Defines how data is loaded and processed.
    2. Semantic information: Adds context and meaning to data columns.
    3. Data transformation: Specifies cleaning and transformation rules.

  3. What is the Judge Agent?

    main

    The JudgeAgent is an extension for the PandasAI library that adds a validation step to the agent pipeline. It evaluates the code generated by an agent against the original user query to ensure accuracy and correctness.

    Note: Using the Judge Agent in production requires a license. Please refer to the license documentation and contact the maintainers if you intend to use it in a production environment.

  4. Use the PandasAI Agent for multi-turn conversations

    main

    The Agent class is designed for multi-turn conversations, unlike the pai.chat() method which is intended for single-session exploratory analysis. An Agent maintains conversation state, allowing it to understand context and follow-up questions.

    To use an agent, instantiate it with your data (e.g., a pandas DataFrame) and use the .chat() method.

    import pandas as pd
from pandasai import Agent

sales_by_country = pd.DataFrame({
    "country": ["United States", "United Kingdom", "France", "Germany", "Italy", "Spain", "Canada", "Australia", "Japan", "China"],
    "sales": [5000, 3200, 2900, 4100, 2300, 2100, 2500, 2600, 4500, 7000],
    "deals_opened": [142, 80, 70, 90, 60, 50, 40, 30, 110, 120],
    "deals_closed": [120, 70, 60, 80, 50, 40, 30, 20, 100, 110]
})

agent = Agent(sales_by_country)
agent.chat('Which are the top 5 countries by sales?')
# The agent remembers this context for the next call
agent.chat('And which one has the most deals?')

  5. How PandasAI works

    main

    PandasAI acts as a translation layer between natural language and executable code. When a user submits a query, the library uses a Large Language Model (LLM) to:

    1. Understand the intent of the natural language query.
    2. Translate that intent into Python code or SQL queries.
    3. Execute the generated code against the provided data source.
    4. Return the results (data or visualizations) to the user.

  8. How Pipelines and Building Blocks work in PandasAI

    main

    Pipelines in PandasAI are used to chain together multiple processing steps, known as Building Blocks. This architecture allows for complex task automation by composing individual logic units into a sequence.

    Core components include:

    • Pipeline: The base class used to chain multiple logic units together.
    • BaseLogicUnit: The fundamental base class that all individual logic units inherit from. Each unit is responsible for performing one specific task within the pipeline.

  9. Achieve determinism in LLM responses

    main

    To ensure reproducible and consistent results (important for testing and debugging), you can configure the LLM's randomness using temperature and seed via pai.config.set().

    • temperature=0: Minimizes randomness by forcing the model to choose the most likely next word. This makes responses predictable but reduces creativity.
    • seed: Sets the initial state of the random number generator for even higher predictability.

    Note on Azure OpenAI: The seed parameter is currently not supported for AzureOpenAI instances. For Azure, use temperature=0 to reduce randomness.

    import pandasai as pai

# Sample DataFrame
df = pai.DataFrame({
    "country": ["United States", "United Kingdom", "France", "Germany", "Italy", "Spain", "Canada", "Australia", "Japan", "China"],
    "gdp": [19294482071552, 2891615567872, 2411255037952, 3435817336832, 1745433788416, 1181205135360, 1607402389508, 1490967855104, 4380756541440, 14631844184064],
    "happiness_index": [6.94, 7.16, 6.66, 7.07, 6.38, 6.4, 7.23, 7.22, 5.87, 5.12]
})

# Configure for maximum determinism
pai.config.set({
   "temperature" : 0,
   "seed" : 26
})

df.chat('Which are the 5 happiest countries?')

  10. How the PandasAI cache works

    main

    PandasAI uses a SQLite database to cache the results of previous queries. This mechanism provides two main benefits:

    1. Performance: Quickly retrieves results for repeated queries without waiting for model generation.
    2. Cost Reduction: Reduces the number of API calls made to the LLM, lowering usage costs.

    The cache is stored in a file named cache.db within a /cache directory. This file is created automatically upon the first query and can be inspected using any standard SQLite client.

  11. How the Semantic Agent works

    main

    The SemanticAgent operates through a two-step process:

    1. Schema Generation: The agent structures data into a schema. By default, it automatically generates a schema based on all dataframes passed to it. Alternatively, you can provide a custom schema during instantiation.
    2. JSON Query Generation: The agent generates a structured JSON query based on the schema. This JSON query is then interpreted and converted into executable Python or SQL code.

  12. How the PandasAI NL Layer works

    main

    The Natural Language (NL) Layer uses generative AI to transform natural language queries into executable code (Python or SQL).

    When you call the .chat() method on a dataframe, PandasAI sends the following context to the LLM:

    1. The user's question.
    2. The table headers.
    3. A sample of 5-10 rows from the Dataframe.

    The LLM generates the relevant code, which is then executed locally on your machine.