DOCSEARCHbyAlgolia
Submit Docs

WhyLabs AI Control Center and whylogs

website·Indexed 35 days ago

https://docs.whylabs.ai/docs/

Documentation for the WhyLabs platform and whylogs, an open-source library for profiling and monitoring ML data pipelines. Includes API references for whylogs core statistics, metrics for NLP and regression, MLflow integration, and guides on Role-based Access Control (RBAC) and SCIM account management.

Tokens
90.3K
Snippets
400
Records
712
Agent score
67%

What's inside WhyLabs

  1. Overview of the Java-based whylogs container

    The whylogs container allows users to generate whylogs profiles without manually integrating the whylogs library into their data pipeline. It accepts data via POST requests or Kafka, converts that data into profiles (aggregating by timestamp), and periodically uploads them to a configured destination (WhyLabs, S3, or local file system).

  2. Overview of WhyLabs AI Control Center capabilities

    WhyLabs AI Control Center is a platform for observing, monitoring, securing, and optimizing AI applications. It provides the following core capabilities:

    Predictive ML Monitoring:

    • Track model performance using built-in or custom metrics (including delayed ground truth).
    • Monitor data quality for model inputs, feature stores, and batch/streaming pipelines.
    • Detect and root cause drift, data quality issues, performance degradation, and model bias.

    Generative AI (LLM) Security and Observability:

    • Implement guardrails using policy rulesets aligned with MITRE ATLAS and LLM OWASP standards via a secure containerized agent and policy manager.
    • Detect and alert on toxicity, prompt injections, PII leakage, malicious activity, and hallucinations.
    • Visualize and debug LLM interactions and RAG events using OpenLLMTelemetry.
    • Track security metrics, costs, and model performance through customizable dashboards.
    • Incorporate user feedback for application improvement.

  3. Overview of Data Segmentation in WhyLabs

    Data segmentation allows users to monitor the health of specific data subgroups that may behave differently from the overall dataset. Segmentation is performed during the profiling stage using whylogs. Once uploaded to WhyLabs, segments appear in the 'Segments' section of the model dashboard, allowing users to drill down into input views, profile comparisons, and alerts specific to that segment. This is particularly useful for identifying anomalies in small subsets of data that might not trigger an alert for the global dataset.

  4. Overview of WhyLabs SCIM V2 Integration

    WhyLabs supports the System for Cross-domain Identity Management (SCIM) V2 protocol, a RESTful standard used to simplify the management of user identities, attributes, and resources. This allows organizations to automate user provisioning, deprovisioning, and synchronization between an identity provider (SCIM Client) and WhyLabs (SCIM Service Provider).

    Availability: SCIM is exclusively available to WhyLabs Enterprise Plan customers. To get started, contact the WhyLabs support team or email support@whylabs.ai.

  5. Overview of WhyLabs Secure guardrails

    WhyLabs Secure is a security layer for AI applications designed to mitigate risks such as hallucinations, data leakage, misuse, and inappropriate content generation. It is LLM-agnostic and compatible with hosted providers (OpenAI, Amazon BedRock, Anthropic, Cohere), open-source models, and custom models.

    Secure operates as a hybrid-SaaS: it requires a containerized agent deployed within the customer's VPC (which includes a dedicated policy API), while the policies are managed via the WhyLabs AI Control Center. It integrates with OpenLLMTelemetry to visualize LLM traces and provide metrics on blocked and flagged interactions. Guardrails are aligned with MITRE ATLAS and LLM OWASP standards.

  6. Overview of WhyLabs SCIM Integration

    The WhyLabs SCIM API allows for the automated provisioning of users and the management of organization memberships across multiple organizations belonging to a single parent organization (the 'account'). This is available exclusively for WhyLabs Enterprise Plan customers.

  7. Overview of whylogs for data profiling

    whylogs is an open-source library used to generate statistical summaries of datasets, known as "profiles". These profiles capture key information about data distributions, missing values, and custom metrics. They are designed to be efficient (compact snapshots), customizable (configurable trackers for different data types), and mergeable (profiles can be combined to represent aggregated data across distributed or streaming systems).

    Common use cases include:

    • Data and concept drift detection
    • ML model performance degradation detection
    • Exploratory data analysis (EDA)
    • Tracking data for ML experiments
    • Data auditing and governance

  8. Overview of openLLMtelemetry for LLM Observability and Guardrails

    openLLMtelemetry is a package built on OpenTelemetry instrumentation designed to provide observability and guardrails for Large Language Models (LLMs). It allows developers to monitor LLM performance, implement safety measures for outputs, and trace data flow across distributed systems (including GenAI agents and workflows). Traces can be exported to WhyLabs or an existing tracing stack.

  9. Overview of the LLM Metric Validator container

    The LLM Metric Validator is a containerized version of the LangKit open-source library. It hosts model assets, dependencies, and metric configurations, exposing HTTP endpoints that LLM applications can call to compute metrics. It can be configured via YAML to specify which built-in or custom metrics to execute and the thresholds used to generate validation reports.

  10. Overview of WhyLabs Secure Guardrails

    WhyLabs Secure provides a set of security capabilities and guardrails for AI applications to mitigate risks including hallucinations, data leakage, misuse, poor user experience, and inappropriate content generation. It aligns with MITRE ATLAS and LLM OWASP standards. While all onboarded LLMs have access to 'Observe' features (monitoring, explainability, performance dashboards), 'Secure' adds specific capabilities for RAG data visualization, trace analysis, and policy management.

  11. Overview of NVIDIA NIM Integration

    NVIDIA NIM is a platform used for managing and deploying AI model APIs. These APIs are designed to be compatible with the OpenAI client, allowing developers to integrate them into existing OpenAI-compatible workflows.

  12. Overview of WhyLabs Monitoring Capabilities

    WhyLabs Observatory provides health monitors for dataset features, model inputs, outputs, and performance. The Monitor Manager tab (available for each model or dataset resource) allows users to track:

    • Data Drift: Distribution similarity and descriptive statistics.
    • Data Quality: Missing values, schema, and cardinality.
    • Concept/Label Drift: Changes in the relationship between inputs and labels.
    • Model Performance: Classification and regression metrics.
    • Categorical Features: Top K values for input features.
    • Data Volume: Input and output volumes.
    • Data Ingestion: Detection of data delivery to a specific model.