Boring Semantic Layer (BSL)

repository·main·Indexed 19 days ago

https://github.com/boringdata/boring-semantic-layer

A lightweight, Ibis-powered semantic layer (version 0.3.16) designed to bridge structured data sources and consumers like LLMs via MCP. It allows developers to define dimensions and measures on top of any Ibis-compatible database engine, supporting advanced analytics such as window functions, Malloy-style bucketing, and contribution analysis using t.all().

Tokens
67.1K
Snippets
203
Records
281
Agent score
65%

What's inside boring-semantic-layer

  1. Use the MCP Semantic Model Server

    main
    The MCP Semantic Model Server is a Model Context Protocol (MCP) server designed for querying structured data via SemanticTable objects. It provides a semantic layer that abstracts raw data into Dimensions (grouping columns), Measures (aggregated metrics), and time-based aggregations. It supports complex operations like joins across multiple tables and time-grain configurations.
  2. Compare BSL, Malloy, and dbt Semantic Layer

    main

    Use the following comparison to determine which semantic layer tool fits your workflow.

    Key Differences

    • BSL (Boring Semantic Layer): Best for Python-first analytics and AI agents. It uses a pure Python API with Ibis expressions, making it highly portable across 20+ databases. It is easy to set up via pip and provides native MCP protocol support for AI/LLM integration.
    • Malloy: Best for complex analytical queries and users wanting a specialized DSL. It features a custom query language with optimized SQL generation and excellent VSCode support. It is Google-backed and focuses on preventing join fan-out through graph-based automatic safety.
    • dbt Semantic Layer: Best for enterprise metrics governance. It relies on YAML configuration and is part of the larger dbt ecosystem. It is designed for analytics engineers and integrates extensively with BI tools like Tableau and Power BI, though it often requires a paid dbt Cloud subscription for full features.

    Comparison Summary

    AspectBSLMalloydbt Semantic Layer
    Language TypePure Python APICustom DSLYAML Configuration
    Query StylePython fluent API (lambdas)Malloy DSL (-> operator)CLI or API calls
    Backend Support20+ databases (via Ibis)7 databasesdbt-supported warehouses
    Setup Complexity⚡ Simple (pip install)Medium (npm/VSCode)Complex (requires dbt project)
    AI/LLM Support✅ MCP protocol nativeNot explicit✅ Native (MetricFlow)
    Primary Use CasePython-first / AI agentsComplex analyticsEnterprise governance
  3. What is Dimensional Indexing and how does it work?

    main

    Dimensional indexing allows you to create a searchable catalog of unique values across your dimensions. It is used for data exploration, building autocomplete features, profiling data distributions, and ranking values by custom metrics.

    When you call the index() method, it returns a standardized table containing the following columns:

    • fieldName: The name of the dimension.
    • fieldValue: The unique value found in that dimension.
    • fieldType: The data type (e.g., string, number).
    • weight: The frequency count of the value, or a custom measure value if a specific measure is provided for ranking.

    This pattern is useful for finding "top cities by revenue" or building type-ahead search suggestions.

    # Example of the resulting index structure concept
    # result = table.index()
    # Columns: [fieldName, fieldValue, fieldType, weight]
  4. Construct compound and nested filters

    main

    To combine multiple filter conditions, use a compound filter object with an operator (AND or OR) and a conditions array containing the sub-filters.

    Compound filters can be nested to create complex logic (e.g., (A AND B) AND C).

    {
        "operator": "AND",
        "conditions": [
            {
                "field": "country",
                "operator": "equals",
                "value": "US"
            },
            {
                "field": "tier",
                "operator": "in",
                "values": ["gold", "platinum"]
            },
            {
                "field": "name",
                "operator": "ilike",
                "value": "%john%"
            }
        ]
    }
  5. How BSL automatically detects chart types

    main

    BSL uses the structure of your query to automatically select the most appropriate chart type:

    • Bar Chart: Triggered by a single categorical dimension and one measure.
    • Time Series (Line Chart): Triggered when a dimension is marked with is_time_dimension: True. BSL applies time-aware formatting.
    • Heatmap: Triggered by two categorical dimensions and one measure.
    • Multi-Series Charts: Triggered when multiple measures are present; BSL automatically applies color encoding based on the measure names.

    To ensure a dimension is treated as a time dimension, configure it in with_dimensions:

    .with_dimensions(
        date={
            "expr": lambda t: t.date.cast("date"),
            "is_time_dimension": True,
            "smallest_time_grain": "TIME_GRAIN_DAY"
        }
    )
  6. Correctly access columns in BSL lambdas

    main

    Column access rules vary depending on the method being used:

    In with_dimensions and with_measures (CRITICAL)

    Access columns directly via the table argument t. Do not use a model prefix.

    • model.with_dimensions(x=lambda t: t.column_name)
    • model.with_dimensions(x=lambda t: t.model.column_name)

    In filter

    You can use a model prefix for joined columns (e.g., t.customers.country), but you must use the exact names provided by get_model().

    # ✅ CORRECT - access columns directly via t
    flights.with_dimensions(x=lambda t: ibis.cases((t.carrier == "WN", "Southwest"), else="Other"))
    flights.with_measures(pct=lambda t: t.flight_count / t.all(t.flight_count) * 100)
    
    # ❌ WRONG - model prefix fails in with_dimensions/with_measures
    flights.with_dimensions(x=lambda t: t.flights.carrier)  # ERROR
  7. Aggregate data by time grain

    main

    To aggregate time-based dimensions, use the time_grain or time_grains parameters instead of using date functions (like .month()) in filters.

    Global Time Grain

    Use time_grain to apply the same grain to all time dimensions in the query. Requirement: You MUST include the time dimension in the dimensions list.

    Available Grains (short or long form):

    • year / TIME_GRAIN_YEAR
    • quarter / TIME_GRAIN_QUARTER
    • month / TIME_GRAIN_MONTH
    • week / TIME_GRAIN_WEEK
    • day / TIME_GRAIN_DAY
    • hour / TIME_GRAIN_HOUR
    • minute / TIME_GRAIN_MINUTE
    • second / TIME_GRAIN_SECOND

    Per-Dimension Time Grains

    Use the time_grains dictionary when different dimensions require different aggregation levels.

    Time Range Filtering

    Use the time_range parameter for preferred time-based filtering. It handles ISO 8601 formats and time zones automatically:

    {
        "start": "2024-01-01T00:00:00Z",
        "end": "2024-12-31T23:59:59Z"
    }
    query_model(
        model_name="orders",
        dimensions=["orders.order_date", "orders.ship_date"],
        measures=["orders.total_sales"],
        time_grains={"orders.order_date": "month", "orders.ship_date": "quarter"}
    )
  8. Calculate percentage of total using the .all() method

    main

    To calculate percentages relative to a grand total (e.g., market share or contribution ratios), use the .all() method within your semantic table definition. The .all(measure) method calculates the total value across all groups, allowing you to define percentage measures directly in the semantic layer. This approach is more efficient than using window functions in post-processing because the percentage logic is embedded in the table definition and works seamlessly across different dimensional breakdowns.

    flights = (
        to_semantic_table(flights_data, name="flights")
        .with_measures(
            flight_count=lambda t: t.count(),
            total_distance=lambda t: t.distance.sum(),
        )
        .with_measures(
            market_share=lambda t: t.flight_count / t.all(t.flight_count) * 100,
            distance_share=lambda t: t.total_distance / t.all(t.total_distance) * 100,
        )
    )
  9. Follow BSL field naming and filtering conventions

    main

    To avoid query errors, adhere to these strict naming and filtering rules:

    Field Naming

    • Use EXACT names from the get_model() output.
    • Joined columns: Use the format t.table_name.column_name (e.g., t.customers.country).
    • Direct columns: Use t.column_name (e.g., t.region).
    • No methods: Never invent methods on columns (e.g., t.customer_id.country() is invalid).

    Filtering Values

    • Never guess filter values: Data often uses codes or IDs (e.g., CA instead of California).
    • Discovery Pattern: Always perform a discovery query (e.g., using group_by) to find valid values before applying a .filter().
  10. Implement the Sessionization pattern

    main

    The sessionization pattern groups sequential time-series events into logical sessions based on inactivity timeouts. The workflow follows these steps:

    1. Identify Boundaries: Use the lag() window function to find the time difference between the current event and the previous event for a specific user. A session start is marked if the gap exceeds a threshold (e.g., 30 minutes) or if it is the user's first event.
    2. Assign Session IDs: Use a cumulative sum (.sum().over(...)) of the session start markers (cast to integers) to generate unique, incrementing IDs for each session per user.
    3. Calculate Metrics: Group by the newly created session_id to calculate session-level metrics like duration, event count, or conversion status.
    4. Aggregate to User Level: Group by user_id to summarize behavior across all sessions (e.g., total sessions, average events per session, conversion rate).
    from ibis import _
    
    # 1. Identify Boundaries & 2. Assign Session IDs
    result = (
        activity_st
        .group_by("user_id", "minute_offset", "page_url", "action")
        .aggregate()
        .mutate(
            prev_minute=lambda t: t.minute_offset.lag().over(
                group_by="user_id",
                order_by=t.minute_offset
            ),
            minutes_since_last=lambda t: t.minute_offset - t.prev_minute,
            is_session_start=lambda t: (t.minutes_since_last > 30) | t.prev_minute.isnull(),
            session_id=lambda t: t.is_session_start.cast("int32").sum().over(
                group_by="user_id",
                order_by=t.minute_offset,
                rows=(None, 0)  # Cumulative sum
            )
        )
    )
  11. Define time dimensions for MCP time-series queries

    main

    To enable LLMs to perform time-based aggregations and range queries via MCP, you must explicitly define time dimensions in your semantic table. This is done by setting is_time_dimension=True and specifying the smallest_time_grain in the dimension configuration.

    Available time grains:

    • second / TIME_GRAIN_SECOND
    • minute / TIME_GRAIN_MINUTE
    • hour / TIME_GRAIN_HOUR
    • day / TIME_GRAIN_DAY
    • week / TIME_GRAIN_WEEK
    • month / TIME_GRAIN_MONTH
    • quarter / TIME_GRAIN_QUARTER
    • year / TIME_GRAIN_YEAR
    from boring_semantic_layer import to_semantic_table
    
    flights = (
        to_semantic_table(flights_data, name="flights")
        .with_dimensions(
            arr_time={
                "expr": lambda t: t.arr_time,
                "description": "Arrival time of the flight",
                "is_time_dimension": True,
                "smallest_time_grain": "TIME_GRAIN_SECOND",
            },
            origin={
                "expr": lambda t: t.origin,
                "description": "Origin airport code"
            },
        )
        .with_measures(
            flight_count={
                "expr": lambda t: t.count(),
                "description": "Total number of flights"
            }
        )
    )