DOCSEARCHbyAlgolia
Submit Docs

Data Formulator

repository·main·Indexed 12 days ago

https://github.com/microsoft/data-formulator

An AI-powered data visualization and exploration tool by Microsoft Research. It enables users to connect to various data sources, use natural language to generate editable charts on an interactive canvas, and perform semantic charting via the Flint visualization language. Features include a plugin system for custom data loaders, conversational database loading in version 0.8 alpha, and support for multiple LLM providers via LiteLLM.

Tokens
70.1K
Snippets
128
Records
253
Agent score
96%

What's inside Data Formulator

  1. Core capabilities of Data Formulator

    main

    Data Formulator is an AI-powered data exploration tool that allows users to interact with data using natural language and visual interfaces.

    Key Workflows:

    • Data Ingestion: Upload structured files (CSV, TSV, XLSX), extract data from screenshots, text blocks, or websites, or connect to databases via connectors.
    • AI-Powered Exploration: Use natural language to ask visualization questions, refine charts, or delegate exploration tasks to agents.
    • Visual Interaction: Edit charts on an interactive canvas, branch analyses into new questions or calculated columns, and create reports to share insights.
    • Semantic Charting: Charts are powered by Flint, an open-source visualization language that compiles semantic specs into polished Vega-Lite, ECharts, and Chart.js visualizations.

  2. Overview of the Streaming Communication Protocol

    main

    All streaming endpoints in Data Formulator use the NDJSON (Newline-Delimited JSON) protocol for communication between the backend and frontend.

    Protocol Specifications:

    • Content-Type: application/x-ndjson
    • Format: Each line is a complete JSON object ending with a newline (\n).
    • Encoding: UTF-8 (with ensure_ascii=False).
    • HTTP Status Codes: Streaming endpoints always return 200. If a pre-check fails before the stream starts, it returns a 200 application/json error envelope. Errors occurring after the stream has been established are sent as events within the stream.

    Prohibitions:

    • Do NOT use the SSE data: prefix.
    • Do NOT mix MIME types.
    • Do NOT return non-JSON lines within the stream.
    {"type": "question", "text": "...", "goal": "...", "tag": "..."}\n
{"type": "warning", "warning": {"message": "..."}}\n
{"type": "error", "error": {"code": "...", "message": "..."}}\n

  3. Two-series and Four-series Color Conventions

    main

    When icons represent charts with color encoding, follow these series conventions:

    Two-series convention

    • Series A: Light blue fill (#76c6ff) / Dark blue stroke (#4a8acb)
    • Series B: Light gray fill (#B8B8B8) / Dark gray stroke (#7a7a7a)

    Four-series convention (e.g., bump charts)

    • Series 1: #76c6ff fill / #4a8acb stroke
    • Series 2: #B8B8B8 fill / #7a7a7a stroke
    • Series 3: #5ba8e0 fill / #3d7ebf stroke
    • Series 4: #d0d0d0 fill / #a8a8a8 stroke

  4. Inspect charts and data before reporting

    main

    To ensure accuracy and avoid inventing data, use the available inspection tools to verify details before composing the report. The report should be grounded in the actual exploration history (the [FOCUSED THREAD] and [OTHER THREADS]).

    Available Inspection Tools:

    • inspect_chart: Allows you to read a chart's encodings, a data sample, and the code that produced it. It also provides a link to the backing table.
    • inspect_source_data: Used to interrogate the full data or table schemas.
    • execute_python_script: Can be used to interrogate full datasets after identifying the backing table via inspect_chart.

  5. Metadata Merging Strategy for UI and Agents

    main

    Data Formulator generates a merged metadata view at runtime without modifying the underlying cache or annotation files. This ensures that user annotations supplement rather than overwrite remote metadata.

    Merging Rules

    • Table Level: display_description = user.description || source.description (used for frontend display).
    • Column Level: display_column_description = user.columns[col].description || source.columns[col].description (used for frontend display).
    • Agent Context: Both source_description and user_description are passed to the Agent. If both exist and differ, the Agent sees them in the format: (source: ... | user: ...). This prevents the Agent from losing original system semantics.
    • Tags and Notes: These are derived exclusively from annotations.

  6. Unified Error Handling Protocol Overview

    main

    Data Formulator (DF) uses a unified error handling protocol to ensure consistency between non-streaming JSON APIs and NDJSON streaming APIs.

    Core Principle: Business and validation errors must return an HTTP 200 status code with a JSON body indicating the error. Only authentication, authorization, and uncontrollable transport errors (like 404 or 500) should use non-200 HTTP status codes.

    Protocol Mapping

    ScenarioHTTP StatusResponse FormatTool/Method
    Non-streaming Success200{"status": "success", "data": ...}json_ok(data)
    Non-streaming Business/Validation Error200{"status": "error", "error": {"code", "message", "retry"}}raise AppError(...)
    Non-streaming Auth Error401 / 403{"status": "error", "error": {"code", "message", "retry"}}raise AppError(...)
    Streaming Preflight Success200application/x-ndjsonN/A
    Streaming Preflight Error200application/json + {"status": "error", ...}stream_preflight_error(...)
    Streaming Fatal Error200NDJSON line: {"type": "error", "error": ...}yield stream_error_event(...)
    Streaming Warning200NDJSON line: {"type": "warning", "warning": ...}yield stream_warning_event(...) or collect_stream_warning()

  7. Configure Data Formulator Authentication Modes

    main

    Data Formulator automatically determines its authentication mode based on the presence of the OIDC_CLIENT_SECRET environment variable:

    ModeTrigger ConditionDescription
    Frontend PKCE (Public Client)OIDC_CLIENT_SECRET is not setThe browser interacts directly with the IdP via PKCE. Tokens reside in the browser. Requires HTTPS or localhost.
    Backend Confidential (Confidential Client)OIDC_CLIENT_SECRET is setData Formulator's server handles authentication. Tokens reside in the server-side session.

    Callback URL: Both modes use the same callback address, which must be registered in your IdP: http(s)://<your-df-host>/auth/callback.

    Minimal Configuration for Backend Mode: If your IdP supports OIDC discovery (via .well-known/openid-configuration), you only need:

    AUTH_PROVIDER=oidc
OIDC_ISSUER_URL=https://your-idp/oauth2
OIDC_CLIENT_ID=df-client-id
OIDC_CLIENT_SECRET=df-client-secret

  8. Understand the Data Formulator Identity System

    main

    Data Formulator uses a namespaced identity system to manage security and data isolation. The identity used determines which workspaces and credentials are accessible.

    • Local mode (127.0.0.1, no auth provider): Identity is local:<os_username>. The server ignores the X-Identity-Id header. Vault and workspaces are tied to the OS user.
    • Anonymous mode (multi-user, no auth provider): Identity is browser:<uuid> generated in the browser's localStorage. The server trusts the client-provided X-Identity-Id header but forces the browser: prefix to prevent spoofing authenticated users.
    • Authenticated mode (auth provider configured): Identity is user:<verified_id> from the auth provider. The X-Identity-Id header is ignored entirely to ensure security.

  9. Core Concepts of Data Connectors

    main

    Understanding the hierarchy of data connection abstractions:

    ConceptDescription
    Loader typeThe registered key in DATA_LOADERS (e.g., postgresql, mysql, superset).
    Connector instanceA specific connection instance (e.g., postgresql:analytics).
    Admin connectorA global, read-only connection configured by admins via connectors.yaml or DF_SOURCES__* env vars.
    User connectorA personal connection created by a user via POST /api/connectors.
    Live loaderAn active ExternalDataLoader instance currently connected under the specific identity, stored in memory.
    Credential vaultAn encrypted storage for sensitive credentials (passwords, tokens) keyed by identity + connector_id.
    External loader pluginA *_data_loader.py file in DF_PLUGIN_DIR used to register new loader types without modifying source code.

  10. Understand Disconnect vs Delete operations

    main

    When managing connector lifecycles, it is critical to distinguish between disconnect and delete operations:

    OperationIn-memory LoaderUser ConfigVault CredentialsUI Card Status
    DisconnectClearedRetainedCleared (current service token/credentials)Retained (can reconnect)
    DeleteClearedDeletedDeletedRemoved

    Lifecycle Details:

    • disconnect: Clears the in-memory loader for the current identity, clears the connector's vault credentials, and calls TokenStore.clear_service_token(connector_id) to clean up the service token. The user's connection definition remains, so the UI card stays visible.
    • delete: Performs everything in disconnect and then removes the user's connector definition. Note that admin connectors cannot be deleted by regular users.
    • OIDC Logout vs Disconnect: OIDC logout only clears the SSO and service token from the current Flask Session; it does not delete the vault credentials saved under the identity + connector_id combination.
    • get-status Requirement: This method must be side-effect free. It should not automatically create a loader or write to the vault just to report connection status.

  11. How the Agent Knowledge System works

    main

    The Agent Knowledge System is composed of three primary components that manage user knowledge, debugging, and experience extraction:

    1. KnowledgeStore: Handles CRUD operations, searching, and parsing of user knowledge files (Rules and Experiences). Located at py-src/data_formulator/knowledge/store.py.
    2. ReasoningLogger: Provides structured JSONL debugging logs for DataAgent sessions. Located at py-src/data_formulator/agents/reasoning_log.py.
    3. ExperienceDistillAgent: Extracts reusable experience documents from experience_context provided by the frontend. Located at py-src/data_formulator/agents/agent_experience_distill.py.

    Knowledge is stored in the user's home directory under <user_home>/knowledge/, organized into rules/ (flat .md files) and experiences/ (up to one level of subdirectories).

  12. Understand DataConnector credential injection

    main

    The DataConnector._inject_credentials() method automatically handles credential injection during connection establishment. The logic follows these steps:

    1. Explicit Tokens: If params already contains access_token or sso_access_token, these are preserved and not overwritten.
    2. Mode Check: It checks the Loader's auth_config().mode.
    3. Default Modes: If the mode is credentials or connection, it returns immediately (these loaders use connection parameters or Vault credentials directly).
    4. Token/SSO/Delegated Modes: For other modes, it calls TokenStore.get_access(source_id) to retrieve the best credential. If a dictionary is returned, it is merged into params. If a string is returned, it is written to params["access_token"].
    5. Fallback: If TokenStore has no credentials, it attempts to inject the raw sso_access_token via auth.identity.get_sso_token() to support loaders that implement their own SSO exchange.