repository·master·Indexed 40 days ago
https://github.com/alibaba/pairecA Go web framework optimized for building online recommendation services. It utilizes JSON configurations and built-in model functionalities to accelerate development. The framework includes tools like pairecmd for project initialization and supports remote model requests via EAS, Seldon, and TensorFlow Serving (TFservingRequest). It also features integrated Prometheus monitoring and a flexible callback hook system for post-recommendation processing.
To use the pairec framework in your Go projects, install the v2 module using go get.
go get github.com/alibaba/pairec/v2pairecmd tool. This tool automates the creation of project structures and service startup.The ConfigLoader implements a polling pattern to ensure the service stays synchronized with the remote configuration server without requiring a restart.
Lifecycle:
NewConfigLoader(configName) sets up the loader with the target config name and a version key (configName + "_version").loadConfigFromConfigServer() retrieves raw data, performs string substitution for environment variables and A/B test parameters, and unmarshals the JSON into a recconf.RecommendConfig object.loopLoadConfig() runs in a background goroutine. Every 10 seconds, it calls loadConfigVersion() to check the current version via the A/B test parameter system.configVersionValue, the loader triggers reloadConfig(config).reloadConfig iterates through all registered subsystems (e.g., mysqldb, redisdb, algorithm, filter, service) and calls their respective .Load(config) methods to apply the new settings in-memory.recconf.UpdateConf(config) is called to update the global configuration state used by the rest of the application.The HaQuery struct allows for complex search configurations. Key components include:
Query: The main search string (required).Cluster: Specifies the cluster or partition.Config: An HaQueryconfigClause defining Start (offset), Hit (count), and Format (e.g., json or xml).Filter: A string used to filter results after the initial query.Sort: A slice of HaQuerySortClause specifying SortKey and SortOrder (+ for ascending, - for descending).Aggregate: A slice of HaQueryAggregateClause for statistical functions like count(), sum(id), max(id), or min(id) on a GroupKey.Distinct: A slice of HaQueryDistinctClause to ensure result diversity (e.g., using DistKey).Kvpairs: A map of key-value pairs passed to feature functions.If PushGatewayUseAliyunCredential is enabled in the Prometheus configuration, Pairec uses an internal aliyunPushGatewayCredential provider. This provider utilizes the Alibaba Cloud default credential chain (supporting environment variables, ECS RAM roles, OIDC, etc.).
It automatically handles STS temporary credentials by formatting the password as {AccessKeySecret}${SecurityToken} to comply with the Alibaba Cloud Prometheus format.
To create a recommendation service, use pairec.Route to map a URL path to a controller that implements the web.Controller interface.
In your main function:
pairec.AddStartHook to register initialization logic.pairec.Route(path, controller) to register your endpoint.pairec.Run() to start the server.Example implementation of a FeedController:
func main() {
pairec.AddStartHook(func() error {
return nil
})
pairec.Route("/api/rec/feed", &controller.FeedController{})
pairec.Run()
}package main
import (
"${BINNAME}/src/controller"
"github.com/alibaba/pairec/v2"
)
func main() {
pairec.AddStartHook(func() error {
return nil
})
pairec.Route("/api/rec/feed", &controller.FeedController{})
pairec.Run()
}To use the HA3Engine client, create a Config object with your connection details and pass it to NewClient. The Config requires an Endpoint and authentication credentials (AccessUserName and AccessPassWord).
Available configuration fields:
Endpoint: The service endpoint.InstanceId: The specific instance ID.Protocol: The communication protocol (e.g., HTTP).AccessUserName: Username for authentication.AccessPassWord: Password for authentication.UserAgent: Custom user agent string.import "github.com/alibaba/pairec/datasource/ha3engine/ha3client"
config := &ha3client.Config{}
config.SetEndpoint("your-endpoint.com")
config.SetInstanceId("your-instance-id")
config.SetAccessUserName("your-username")
config.SetAccessPassWord("your-password")
client, err := ha3client.NewClient(config)
if err != nil {
// handle error
}The Pairec service is configured using a JSON file (typically config.json). Key configuration sections include:
RunMode: Execution mode (e.g., product).ListenConf: Network settings including HttpAddr and HttpPort (default 8000).ABTestConf: Configuration for A/B testing with Host and Token.RecallConfs: List of recall configurations. Each entry requires a Name, RecallType, and RecallCount.SortNames and FilterNames: Mapping of default sorting and filtering strategies.SceneConfs: Scene-specific configurations. For example, a home_feed scene can define specific RecallNames for its default configuration.LogConf: Logging settings including RetensionDays, DiskSize, and LogLevel (e.g., INFO).HologresConfs, KafkaConfs, RedisConfs: Connection settings for external data stores.{
"RunMode": "product",
"ListenConf": {
"HttpAddr": "",
"HttpPort": 8000
},
"ABTestConf": {
"Host": "",
"Token": ""
},
"RecallConfs": [
{
"Name":"mock_recall",
"RecallType":"MockRecall",
"RecallCount":200
}
],
"SceneConfs": {
"home_feed":{
"default":{
"RecallNames":["mock_recall"]
}
}
},
"LogConf": {
"RetensionDays": 3,
"DiskSize": 20,
"LogLevel": "INFO"
}
}Pairec integrates with Prometheus via the github.com/alibaba/pairec/v2/middleware/prometheus package. Monitoring is enabled by setting recconf.Config.PrometheusConfig.Enable to true.
Key configuration options available in recconf.Config.PrometheusConfig:
Subsystem: A string to define the Prometheus subsystem.ReqDurBuckets: Buckets for request duration histograms.ReqSizeBuckets: Buckets for request size histograms.RespSizeBuckets: Buckets for response size histograms.PushGatewayURL: The URL for the Prometheus Pushgateway.PushGatewayToken: The token used for Pushgateway authentication.PushIntervalSecs: The interval at which metrics are pushed.Job: The job name for the push (defaults to recconf.Config.RunMode or the PAIREC_ENVIRONMENT env var if not specified).PushGatewayUseAliyunCredential: If true, uses Alibaba Cloud credentials for the Pushgateway.PushGatewayUsername / PushGatewayPassword: Used for Basic Auth if not using Aliyun credentials.The behavior of the CallBackHookFunc is controlled by two main configuration areas within the RecommendContext.Config:
SceneConfs)Used to determine if the callback should be triggered at all for a specific scene.
AutoInvokeCallBack: A boolean flag in the default category of a scene's configuration that enables/disables the hook.AutoInvokeCallBackRate: An integer (0-100) representing the percentage of requests that should trigger the callback.CallBackConfs)Used to shape the data sent to the callback service for a specific scene.
ItemSize: Limits the number of items sent in the ItemList.ItemSizeRate: A percentage (0-100) used to downsample the item list. If set, the system selects a subset of items using a randomized step-based approach.UseUserFeatures: A boolean determining whether to use user.MakeUserFeatures2() or to extract features from the features parameter in the context.The resulting web.CallBackParam contains:
SceneId: The current scene string.RequestId: The RecommendId from the context.Uid: The string representation of the user ID.Features: A map of user features.ItemList: A slice of maps, where each map contains:item_id: The item ID.score: The item score.algo_scores: A clone of the item's algorithmic scores.item.GetCloneFeatures().The log package provides simple utilities for printing formatted status messages to the console during CLI execution. It uses color coding to distinguish between success and failure states.
Info(message string): Prints a message prefixed with a green [SUCCESS] tag. This is used for non-terminal informational messages.Error(message string): Prints a message prefixed with a red [FAILED] tag and immediately terminates the process with exit code -1. This should be used for unrecoverable errors.package main
import "github.com/alibaba/pairec/commands/log"
func main() {
// Prints [SUCCESS] message in green
log.Info("Operation completed successfully")
// Prints [FAILED] message in red and exits with -1
log.Error("An unexpected error occurred")
}