Refinery Documentation

Refinery is a tail-based sampling proxy for Honeycomb. Unlike head sampling, which makes a decision at the start of a trace, Refinery examines entire traces to make intelligent sampling decisions. This allows you to keep traces based on their content (e.g., keeping all traces that contain a 500 error or specific cache status) while dropping less useful data to manage volume.

Refinery supports several sampling techniques:

• Dynamic sampling: Automatically adjusts sampling rates based on the frequency of unique values in a trace field (e.g., sampling 100% of 5xx errors but only 0.1% of 2xx successes).
• Rules-based sampling: Defines specific sampling rates for well-known conditions (e.g., 'keep 100% of traces with an error').
• Throughput-based sampling: Maintains a fixed upper bound for spans per second by dynamically adjusting the sampling rate.
• Deterministic probability sampling: Applies decisions based solely on the trace ID, ensuring consistent sampling without inspecting trace content.

The FieldList defines the fields used to form the key for the DynamicSampler. The combination of these field values determines how traces are grouped and sampled.

Best Practices for Field Selection

• Goal: Choose fields that have consistent values for 'boring' traffic and unique values for 'interesting' traffic (outliers).
• Good Choices: Fields like HTTP status code or HTTP endpoint combined with HTTP method. Using the root. prefix (e.g., root.http.response.status_code) is recommended to reduce cardinality by focusing only on the root span.
• Bad Choices: High-cardinality fields like k8s.pod.id (makes every trace unique, resulting in 100% sampling) or single fields like HTTP endpoint (not unique enough to catch outliers like 500 errors).

Key Management

• MaxKeys: Limits the number of distinct keys tracked. Once reached, new keys are not added to the sample rate map, but existing ones continue to be counted. Defaults to 500.
• UseTraceLength: If true, the number of spans in the trace is included in the key. Set to true only if trace length variations are a useful indicator for your analysis.

Virtual fields are prefixed with ?. and allow you to make sampling decisions based on properties of the entire trace rather than individual spans.

Currently, only one virtual field is supported:

• ?.NUM_DESCENDANTS: The total number of child elements contained within a trace.

Example: Drop single-span traces

    - Name: drop single-span traces
      Drop: true
      Conditions:
        - Operator: has-root-span
          Value: true
        - Field: "?.NUM_DESCENDANTS"
          Operator: =
          Value: 1
          Datatype: int

Example: Drop large traces (more than 1000 spans)

Rules:
    - Name: Drop any big traces
      Drop: true
      Conditions:
        Field: "?.NUM_DESCENDANTS"
        Operator: ">="
        Value: 1000
        Datatype: int

Note: For efficiency, combine this with the Traces.SpanLimit configuration set to the same value to reduce memory load.

PeerManagement defines how Refinery instances in a cluster locate and communicate with each other.

Peer Management Types

• file: Refinery uses a static list of peers provided in the Peers field of the configuration file. This prevents the use of publish/subscribe mechanisms for propagating peer lists, stress levels, or configuration changes.
• redis: (Recommended) Refinery uses Redis-based Publish/Subscribe to propagate peer lists, stress levels, and configuration changes quickly. If using redis, you must also configure RedisPeerManagement.

Identity and Discovery

• Identifier: The identifier used when registering with peers. Defaults to the local hostname. Can be overridden with an IP address (e.g., 192.168.1.1).
• IdentifierInterfaceName: If you need to use IPs as identifiers and cannot rely on hostname resolution, specify the network interface (e.g., eth0). Refinery will use the first unicast address on that interface.
• UseIPV6Identifier: If true and IdentifierInterfaceName is set, Refinery will use the first IPv6 unicast address found instead of IPv4.
• Peers: (Only used when Type is "file") A list of peer addresses in the format scheme://host:port (e.g., http://192.168.1.11:8081). This list is eligible for live reload.

Note: Peer management settings are not eligible for live reload, except for the Peers list when using file type.

# Example Peer Management using file
PeerManagement:
  Type: "file"
  Identifier: "192.168.1.1"
  Peers:
    - "http://192.168.1.11:8081"
    - "http://192.168.1.12:8081"

Refinery uses memory settings to prevent crashes during traffic bursts. You should configure memory limits to ensure the process stays within its allocated resources.

There are two ways to manage memory allocation:

1. Using AvailableMemory: Set the total system memory available to the Refinery process. This is recommended for containerized environments. If set, you must not define MaxAlloc.
2. Using MaxAlloc: Set a hard limit on the number of bytes the collector can allocate. If set, you must not define AvailableMemory.

Additionally, you can use MaxMemoryPercentage to set a target maximum percentage (1-100) of memory that should be allocated. If the allocation exceeds this percentage, traces will be ejected from the cache early to reduce memory usage.

# Example using AvailableMemory (Recommended for Kubernetes)
Collection:
  AvailableMemory: 4.5Gb
  MaxMemoryPercentage: 80

# OR Example using MaxAlloc
Collection:
  MaxAlloc: 2GiB
  MaxMemoryPercentage: 75

Refinery uses PeerManagement to allow cluster members to locate each other and propagate state (like stress levels and configuration changes).

Peer Management Types

• file (Default): Refinery reads peer addresses from the Peers list in the configuration file. It does not use a publish/subscribe mechanism.
• redis (Recommended): Refinery uses Redis Pub/Sub to propagate peer lists and state changes much faster. Requires RedisPeerManagement configuration.

Peer Identification

• Identifier (string): The specific ID used to register with peers (e.g., an IP address like 192.168.1.1). Overrides IdentifierInterfaceName.
• IdentifierInterfaceName (string): A network interface (e.g., eth0) used to find a local hostname. Refinery will use the first unicast address found on this interface.
• UseIPV6Identifier (bool): If true, Refinery uses the first IPv6 unicast address found on the specified interface instead of IPv4.

Peer List (for file type)

• Peers (stringarray): A list of peer addresses in the format scheme://host:port (e.g., http://192.168.1.11:8081). This setting is eligible for live reload.

# Example Peer Management using file
PeerManagement:
  Type: "file"
  Peers: ["http://192.168.1.11:8081", "http://192.168.1.12:8081"]

Rules-based samplers use conditions to decide if a rule matches. Conditions are evaluated in order: the first condition that does not match causes the rule to fail. If all conditions match, the rule matches. If no conditions are provided, the rule always matches.

Field Selection

• Field: A single field name to check. Comparison is case-sensitive. If the field is missing, the condition fails.
  • Root Span Prefix: Use the root. prefix (e.g., root.http.status) to evaluate the field specifically against the root span.
  • Note: When using root. with a not-exists operator, you must include the has-root-span: true condition in the rule to avoid false negatives if the root span is missing due to a TraceTimeout.
• Fields: An array of field names. The first field in the array that contains a value is used for the condition. If no fields are present, the condition fails.

Comparison and Types

• Operator: The comparison logic.
  • Options: =, !=, >, <, >=, <=, starts-with, contains, does-not-contain, exists, not-exists, has-root-span, matches, in, not-in.
  • Warning on Negative Operators: When using Scope: trace, a negative operator (like !=) returns true if any single span in the trace matches the condition. To avoid unexpected behavior, use Scope: span when employing negative operators.
• Value: The value to compare against. For in and not-in operators, this can be a list of values of the same datatype.
• Datatype: Forces a specific type for comparison. This is highly recommended to avoid ambiguity (e.g., when a status code might be a string in one environment and an integer in another).
  • Options: string, int, float, bool

The EMADynamicSampler (Exponential Moving Average Dynamic Sampler) is an advanced sampler that attempts to maintain a target sample rate by weighting rare and frequent traffic differently. It is an improvement over the standard DynamicSampler and is recommended for most use cases.

Unlike a simple dynamic sampler that computes rates based on periodic samples, the EMADynamicSampler maintains an EMA of counts seen per key and adjusts the average at regular intervals. This allows it to adapt to traffic patterns more intelligently.

Key Behaviors:

• Adaptability: Controlled by the Weight parameter. Higher weights make the sampler adapt faster to traffic changes; lower weights make it more consistent and less sensitive to bursts.
• New Keys: Keys not already in the EMA are always sampled at a rate of 1 (100%).
• Logarithmic Scaling: Frequent keys are sampled on a logarithmic curve to ensure they don't overwhelm the sampling budget while still trending toward the GoalSampleRate.
• Trace Consistency: Sampling is calculated from the trace ID, ensuring all spans within the same trace are sampled or discarded together.

# Example conceptual configuration for EMADynamicSampler
name: EMADynamicSampler
GoalSampleRate: 30
Weight: 0.5
AdjustmentInterval: 30s

The Exponential Moving Average (EMA) Dynamic Sampler (EMADynamicSampler) is an advanced sampling strategy that attempts to maintain a target average sample rate by weighting rare and frequent traffic differently.

Unlike a simple DynamicSampler which uses periodic samples, the EMADynamicSampler maintains an EMA of counts seen per key and adjusts the average at regular intervals. This makes it more robust for varying traffic patterns.

Key Behaviors:

• Adaptability: Controlled by the Weight parameter. Higher weights allow the sampler to adapt quickly to traffic changes; lower weights provide more consistency and resistance to bursts.
• Logarithmic Scaling: Keys that occur more frequently are sampled on a logarithmic curve.
• Guaranteed Representation: Every key is represented at least once in any given window, and frequent keys have their sample rates increased proportionally to trend toward the GoalSampleRate.
• New Keys: Keys not already present in the EMA always start with a sample rate of 1 (100% sampling).

The EMAThroughputSampler (Exponential Moving Average Throughput Sampler) is a recommended sampler for most throughput-based use cases. It attempts to achieve a target throughput (spans per second) by weighting rare and frequent traffic differently. It maintains an EMA of counts seen per key and adjusts sample rates at regular intervals.

Key behaviors:

• Adaptability: Uses a Weight to balance recent observations against historical data. Higher weights adapt faster to traffic changes; lower weights provide more consistency.
• Logarithmic Scaling: Frequent keys are sampled on a logarithmic curve.
• Guaranteed Representation: Every key is represented at least once in any given window, and frequent keys have their sample rate increased proportionally to trend towards the goal throughput.
• New Keys: New keys not present in the EMA always start with a sample rate of 1.

/* Note: This is a conceptual description of the EMAThroughputSampler logic. 
   Configuration parameters are detailed in the individual reference records. */