QueryChat
repository·main·Indexed 42 days ago
https://github.com/posit-dev/querychatA tool for natural language exploration of tabular data that combines Large Language Models (LLMs) with SQL to ensure safe and reliable querying. It supports Python (with Gradio, Dash, and Streamlit) and R (with Shiny), allowing users to translate natural language into SQL queries executed against data sources like pandas, polars, SQLite, PostgreSQL, and MySQL. It features a schema retrieval system to reduce hallucinations and provides a read-only interface for data filtering, aggregation, and visualization.
What's inside QueryChat
- QueryChat is a tool designed for safe and reliable natural language exploration of tabular data. It works by leveraging SQL and Large Language Models (LLMs) to translate natural language queries into data explorations.
Apply pre-aggregation vs post-aggregation filters in SEMANTIC_VIEW()
mainWhen using
SEMANTIC_VIEW(), you can apply filters in two different stages of the query lifecycle:- Inside the function (Pre-aggregation): Filters the base data before metrics are computed. Use this to limit the scope of the calculation (e.g., only including specific account types in a revenue sum).
- Outside the function (Post-aggregation): Filters the results after metrics have been computed. Use this to filter the final aggregated values (e.g., showing only rows where revenue exceeds a certain threshold).
Examples
Pre-aggregation filter (Inside):
SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD WHERE REF_ENTITIES.ACC_TYPE_CD = 'EXT' )Post-aggregation filter (Outside):
SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD ) WHERE NET_REVENUE > 1000000-- Pre-aggregation: only include 'EXT' accounts in the calculation SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD WHERE REF_ENTITIES.ACC_TYPE_CD = 'EXT' ) -- Post-aggregation: compute all, then filter results SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD ) WHERE NET_REVENUE > 1000000Use REMAPPING for Statistical Layers
mainSome
DRAWlayers compute statistics. To access these computed values, use theREMAPPINGkeyword within theDRAWclause.Layer Computed Stats Default Remapping bar(y unmapped)count,proportioncount AS yhistogramcount,densitycount AS ydensitydensity,intensitydensity AS yviolindensity,intensitydensity AS offsetsmoothintensityintensity AS yboxplotvalue,typevalue AS yExamples:
Density histogram:
VISUALISE FROM products DRAW histogram MAPPING price AS x REMAPPING density AS yBar showing proportion:
VISUALISE FROM sales DRAW bar MAPPING region AS x REMAPPING proportion AS yOverlay histogram and density:
VISUALISE FROM measurements DRAW histogram MAPPING value AS x SETTING opacity => 0.5 DRAW density MAPPING value AS x REMAPPING intensity AS y SETTING opacity => 0.5-- Density histogram (instead of count) VISUALISE FROM products DRAW histogram MAPPING price AS x REMAPPING density AS y -- Bar showing proportion VISUALISE FROM sales DRAW bar MAPPING region AS x REMAPPING proportion AS y -- Overlay histogram and density on the same scale VISUALISE FROM measurements DRAW histogram MAPPING value AS x SETTING opacity => 0.5 DRAW density MAPPING value AS x REMAPPING intensity AS y SETTING opacity => 0.5 -- Violin plot SELECT department, salary FROM employees VISUALISE department AS x, salary AS y DRAW violinHow querychat works: LLM to SQL translation
mainquerychatuses Large Language Models (LLMs) to translate natural language into SQL queries. To ensure accuracy without compromising privacy, it follows these principles:- Schema Retrieval: Instead of sending raw data to the LLM,
querychatuses a schema retrieval tool. The LLM calls this tool on demand to learn about table structures, including column names, types, ranges, and categorical values. - SQL Execution: When the LLM generates a query,
querychatexecutes it against a SQL database (using DuckDB by default) to retrieve precise results. - Contextual Interpretation: The query results are provided back to the LLM so it can interpret and summarize the findings for the user.
Key Design Benefits:
- Reliable: Results come from actual database execution, not LLM-generated summaries, reducing hallucinations.
- Safe: Tools are read-only by design. To ensure production safety, it is recommended to use read-only database permissions.
- Reproducible: The generated SQL can be exported and run in other environments.
- Schema Retrieval: Instead of sending raw data to the LLM,
Use Semantic Views instead of raw tables
mainWhen working with databases that support Semantic Views, you should prefer them over querying raw tables directly. Semantic Views provide a curated layer that encodes business logic, pre-defined metrics, dimensions, and relationships. Using these views ensures consistent and accurate results by applying certified calculation rules (such as discount factors or status code filters) that raw tables lack.
To benefit from these certified definitions when generating SQL, use the
SEMANTIC_VIEW()table function where appropriate.-- Example pattern for using semantic views in SQL SELECT * FROM SEMANTIC_VIEW('NET_REVENUE');Filter data with pre-aggregation vs post-aggregation WHERE clauses
mainThe
SEMANTIC_VIEW()function supports two types of filtering, which behave differently regarding when the filter is applied in the computation lifecycle:- Inside the function (Pre-aggregation): The
WHEREclause inside theSEMANTIC_VIEW()parentheses filters the base data before metrics are computed. Use this to limit the scope of the calculation (e.g., only including specific account types in a sum). - Outside the function (Post-aggregation): The
WHEREclause after the function call filters the results after metrics have been computed. Use this to filter the final aggregated rows (e.g., finding all dimensions where the total revenue exceeds a threshold).
Examples
Pre-aggregation (Filter base data):
SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD WHERE REF_ENTITIES.ACC_TYPE_CD = 'EXT' )Post-aggregation (Filter results):
SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD ) WHERE NET_REVENUE > 1000000-- Pre-aggregation: only include 'EXT' accounts in the calculation SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD WHERE REF_ENTITIES.ACC_TYPE_CD = 'EXT' ) -- Post-aggregation: compute all, then filter results SELECT * FROM SEMANTIC_VIEW( MODEL_NAME METRICS T_DATA.NET_REVENUE DIMENSIONS REF_ENTITIES.ACC_TYPE_CD ) WHERE NET_REVENUE > 1000000- Inside the function (Pre-aggregation): The
Security and Performance considerations for querychat
mainSecurity
- Read-Only Access: Only
SELECTqueries are allowed.querychatdoes not performINSERT,UPDATE, orDELETEoperations. It is highly recommended to use a database user with read-only permissions. - Transparency: All SQL queries generated are visible to users.
- Permissions: Ensure users only have access to the intended tables.
Performance
- Indexing: Use appropriate database indexes on columns frequently used in queries.
- Row Limits: Consider limiting row counts for very large tables.
- Caching:
querychatcaches schema information to avoid repeated metadata queries. - Connection Pooling: For production environments, consider using connection pooling.
- Read-Only Access: Only
Interpret variable nuances in the Titanic dataset
mainWhen analyzing the Titanic dataset, note the following semantic details:
pclass(Socio-economic status):- 1st class = Upper class
- 2nd class = Middle class
- 3rd class = Lower class
age:- Ages less than 1 year are represented as fractions.
- Estimated ages are denoted with a
.5suffix (e.g.,xx.5).
sibsp(Siblings/Spouses): Includes brothers, sisters, stepbrothers, stepsisters, husbands, and wives. Mistresses and fiancés are excluded.parch(Parents/Children): Includes mothers, fathers, daughters, sons, stepdaughters, and stepsons. Note that some children traveling only with a nanny will have aparchvalue of0.
Enable visualization in querychat
mainIf a user asks for a plot or chart and visualization is not working, it may not be enabled. To enable visualization, a developer must:
- Install the package with the visualization extra:
pip install querychat[viz] - Add
"visualize"to thetoolsparameter in the configuration.
pip install querychat[viz]- Install the package with the visualization extra:
SQL Query Guidelines for querychat
mainWhen generating SQL queries for the
querychatchatbot, follow these structural and naming rules to ensure compatibility with the underlying query engine and the parent query execution flow.Structural Rules
- No trailing semicolons: Never end a query with a semicolon (
;). The system wraps your query, and a semicolon will cause syntax errors. - Single statement only: Return exactly one
SELECTstatement. Do not provide multiple statements. - No procedural or meta statements: Do not use
EXPLAIN,SET, variable declarations, transaction controls (BEGIN,COMMIT,ROLLBACK), DDL (CREATE,ALTER,DROP),INTOclauses, or locking hints (FOR UPDATE,FOR SHARE).
Column Naming Rules
- Alias all computed/derived columns: Every expression that is not a simple column reference must have an explicit alias.
- Ensure unique column names: The result set must not contain duplicate column names, especially when performing joins.
- Avoid
SELECT *with JOINs: Explicitly list columns to prevent duplicate names and ensure a predictable schema. - Avoid reserved words as unquoted aliases: If an alias is a reserved word, quote it appropriately for the database dialect.
- No trailing semicolons: Never end a query with a semicolon (
Access computed statistics with REMAPPING
mainStatistical layers (like
bar,histogram,density,violin,smooth, andboxplot) compute new values. Use theREMAPPINGkeyword within aDRAWclause to access these computed statistics.Layer Computed Stats Default Remapping bar(y unmapped)count,proportioncount AS yhistogramcount,densitycount AS ydensitydensity,intensitydensity AS yviolindensity,intensitydensity AS offsetsmoothintensityintensity AS yboxplotvalue,typevalue AS yExample: Using density instead of count in a histogram
-- Density histogram (instead of count) VISUALISE FROM products DRAW histogram MAPPING price AS x REMAPPING density AS y -- Bar showing proportion VISUALISE FROM sales DRAW bar MAPPING region AS x REMAPPING proportion AS y -- Violin plot SELECT department, salary FROM employees VISUALISE department AS x, salary AS y DRAW violinCreate clickable UI suggestions
mainTo provide users with actionable next steps, use
<span class="suggestion">tags. These render as clickable buttons in the UI.Formats
1. List Format (Recommended) Use explicit HTML
<ul>and<li>tags for reliability. This is the most common pattern for follow-up options at the end of a response.<ul> <li><span class="suggestion">Show me examples of …</span></li> <li><span class="suggestion">Explain how …</span></li> </ul>2. Inline Format Use within prose for secondary options. These render as clickable text links.
You might want to <span class="suggestion">explore the advanced features</span>.Guidelines
- Always provide suggestions at the start of a conversation, when beginning a new exploration, or after completing a topic.
- Write complete, natural prompts (e.g., "Show records from the year 2023" instead of "2023").
- Only suggest actions that can be performed with available tools.
- Avoid generic phrases like "If you'd like to..." or "Would you like to explore...".
- Never use nested lists; use headings to group suggestions instead.
##### Analyze the data <ul <li><span class="suggestion">What's the average …?</span></li> <li><span class="suggestion">How many …?</span></li> </ul>