DOCSEARCHbyAlgolia
Submit Docs

TorchServe Documentation

repository·master·Indexed 26 days ago

https://github.com/pytorch/serve

A flexible tool for serving and scaling PyTorch models in production, supporting on-prem, cloud (AWS, GCP), Kubernetes, and Large Language Models (LLMs). This documentation covers benchmarking with Locust and Apache Bench, building and installing release and nightly binaries for PyTorch, Torch-Model-Archiver, and Torch-Workflow-Archiver, and managing Docker image tagging and deployment.

Tokens
141.7K
Snippets
479
Records
753
Agent score
87%

What's inside TorchServe

  1. Overview of TorchServe gRPC APIs

    master

    TorchServe provides gRPC APIs for both inference and management operations.

    Default Ports (localhost):

    • Inference API: 7070
    • Management API: 7071

    Inference API Capabilities:

    • Ping: Retrieves the health status of the server.
    • Predictions: Returns predictions from the served model.
    • StreamPredictions: Provides server-side streaming predictions (useful for high-latency tasks like LLM token generation).

    Management API Capabilities:

    • RegisterModel: Serves a model or model version.
    • UnregisterModel: Removes a specific model version to free resources.
    • ScaleWorker: Adjusts the number of workers for a model version.
    • ListModels: Queries default versions of currently registered models.
    • DescribeModel: Retrieves runtime status details for a model's default version.
    • SetDefault: Sets a specific registered version as the default.

    Security Note: Inference requests require an Inference token, and Management requests require a Management token, unless token authorization is explicitly disabled. Current gRPC implementation does not support workflows.

  2. Overview of KServe Wrapper

    master

    The KServe wrapper enables integration between TorchServe and KServe. It consists of three primary components:

    1. __main__.py: Parses the model snapshot from config.properties provided by KServe and passes parameters (inference address, management address, and model address) to KServe to handle requests.
    2. TorchserveModel.py: Contains methods to handle requests and responses between TorchServe and KServe.
    3. TSModelRepository.py: Handles the initialization of parameters passed to TorchserveModel.py.

    Note: This project is no longer actively maintained.

  3. Overview of TorchServe

    master
    TorchServe is a model serving tool for production inference. It wraps PyTorch deep learning models in REST-based APIs and provides a command-line interface to manage model serving. It supports custom services for specialized inference logic and the ability to serve multiple models simultaneously.

  4. Understand TorchServe Architecture and Terminology

    master

    TorchServe is a multi-model inferencing framework that manages model lifecycles, inference requests, and logging. Key architectural components include:

    • Frontend: The Java-based request/response handling component. It manages client communication and model lifecycles.
    • Model Workers: Processes responsible for executing actual inference on models.
    • Model: Can be script_module (JIT saved models) or eager_mode_models. Models include custom pre/post-processing logic and artifacts like state_dicts. They are loaded from a Model Store (a directory of loadable models) or cloud storage.
    • Plugins: Custom endpoints, authentication/authorization (authn/authz), or batching algorithms that can be added at startup.
    • Model Store: A directory containing all loadable models.

  5. TorchServe Overview and Maintenance Status

    master

    TorchServe is a performant, flexible, and easy-to-use tool designed for serving PyTorch models in production environments.

    ⚠️ Notice: Limited Maintenance This project is no longer actively maintained. While existing releases are available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.

  6. Maintenance Status of TorchServe

    master

    ⚠️ Notice: Limited Maintenance

    TorchServe is no longer actively maintained. While existing releases are available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.

  8. Important notice regarding TorchServe maintenance status

    master

    ⚠️ Notice: Limited Maintenance

    TorchServe is no longer actively maintained. While existing releases are available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that security vulnerabilities may not be addressed.

  9. Types of TorchServe plugins

    master

    You can extend TorchServe behavior by creating two main types of plugins:

    1. Endpoint APIs: Add custom APIs to customize related behaviors. Supported API types include:
      • management api
      • inference api
      • metric api
    2. Snapshot Serializer: Override the default file-based serializer. For example, you can implement a serializer to enable snapshot serialization to services like AWS DynamoDB.

  10. Understand TorchServe log types

    master

    TorchServe provides two primary types of logs:

    1. Access Logs: These track access patterns to the server. They are stored in {LOG_LOCATION}/access_log.log by default. Each entry typically includes the timestamp, remote host, HTTP method, endpoint, protocol, status code, and request duration in milliseconds.
    2. TorchServe Logs: These collect all logs from the TorchServe core and the backend workers (your custom model code). By default, these logs capture everything above the DEBUG level and are stored in {LOG_LOCATION}/ts_log.log.

  12. Understand the TorchServe REST API structure

    master

    TorchServe provides a RESTful API compliant with the OpenAPI specification 3.0. Upon startup, TorchServe initializes several distinct web services. You can use swagger codegen to generate client-side code for Java, Scala, C#, or Javascript based on these specifications.

    The available API services are:

    • Inference API: For performing model inference.
    • Management API: For managing models (e.g., registering, deleting).
    • Metrics API: For accessing service metrics.
    • Workflow Inference API: For workflow-based inference.
    • Workflow Management API: For workflow management.

    Note: This project is no longer actively maintained. There are no planned updates, bug fixes, or security patches.