kvproto

The DB9 protocol does not introduce a new gRPC service. Instead, it reuses the existing TiKV coprocessor.Request envelope to leverage existing routing, region clipping, and admission control.

To transport a DB9 request, you must configure the outer coprocessor.Request as follows:

• tp: Set to REQ_TYPE_DB9_DAG.
• ranges: Provide concrete, region-clipped key ranges.
• data: Provide the serialized Db9DagRequest payload.

This separation allows the outer envelope to manage physical routing while the inner DB9 payload manages logical scan shapes and execution metadata.

coprocessor.Request {
  tp: REQ_TYPE_DB9_DAG
  ranges: [region_clipped_ranges]
  data: serialized_Db9DagRequest
}

When modifying the backup metadata wire schema in proto/brpb.proto (specifically for backup.BackupMeta or any messages reachable from it), you must increment the BackupSchemaVersion to ensure compatibility.

The version constant is located in pkg/brpb/backup_schema_version.go.

The simplest way to compile the protobufs without managing local dependencies is via Docker.

1. Build the Docker image:

   ./scripts/docker-build.sh

2. Run the compilation command (e.g., for Go):

   ./scripts/docker-run.sh make go

kvproto is a repository containing the Protocol Buffer files used by TiKV. Developers use it to manage the schema definitions for TiKV's communication protocols and to generate language-specific bindings for Go and Rust.

Workflow for updating or adding protocols:

1. Add/Modify Protos: Write new .proto files in the proto/ folder.
2. Sync raft-rs (if needed): If you need to update raft-rs, download the corresponding .proto file from the raft-rs repository and overwrite the existing file in the include/ folder.
3. Generate Code: Run make to generate the language bindings.
   • Go code is generated in the pkg/ folder.
   • Rust code is generated in the src/ folder.
4. Update Dependencies: Update any dependent projects to use the newly generated code.

To prevent IDE-induced diffs and maintain consistency, kvproto uses buf format. The formatter is automatically downloaded into the ./bin directory (which is gitignored) when required.

• Check formatting: Run make proto-fmt-check. This is also executed automatically during make check.
• Apply formatting in-place: Run make proto-fmt.

If make check fails due to formatting issues, run make proto-fmt and then rerun make check.

The Db9SelectResponse is the unary response payload returned by the coprocessor. It contains:

• rows: The result rows. For aggregate pushdown, rows are laid out as [group_keys..., partial_states...].
• stats: A Db9ExecStats object reporting:
  • scanned_keys
  • scanned_rows
  • decoded_rows
  • returned_rows
• warnings: Any execution warnings.

Note: The current implementation is unary only. Streaming responses are not yet supported and will be rejected by the executor.

The protocol uses Db9TypeKind to model the DB9 type namespace and Db9Value to carry actual data.

Db9Value supports the following wire values:

• null_value
• bool_value
• int32_value
• int64_value
• float64_value
• text_value
• bytes_value
• timestamp_value (encoded as milliseconds since Unix epoch)
• timestamptz_value (encoded as milliseconds since Unix epoch)

enum Db9TypeKind { ... }

message Db9Value {
  oneof value {
    bool bool_value = 1;
    int32 int32_value = 3;
    int64 int64_value = 4;
    float64 float64_value = 5;
    string text_value = 6;
    bytes bytes_value = 7;
    int64 timestamp_value = 8;    // ms since epoch
    int64 timestamptz_value = 9;  // ms since epoch
    ... 
  }
}

When using the prost-codec feature, the UploadRequest type provides helper methods to manage its chunk field, which can contain either raw data or SST metadata.

• set_data(v: Vec<u8>): Sets the chunk as upload_request::Chunk::Data.
• get_data() -> &[u8]: Returns the data slice, or an empty slice if no data is present.
• set_meta(v: SstMeta): Sets the chunk as upload_request::Chunk::Meta.
• get_meta() -> &SstMeta: Returns a reference to the metadata, or a default reference if none is present.
• has_meta() -> bool: Returns true if the chunk contains metadata.

// Example usage of UploadRequest helpers
let mut request = UploadRequest::default();
request.set_data(vec![1, 2, 3]);
assert_eq!(request.get_data(), &[1, 2, 3]);

request.set_meta(some_sst_meta);
assert!(request.has_meta());

When using the prost-codec feature, the Error type provides a way to attach and retrieve specific error details including RegionError, KvError, and ClusterIdError.

Setting Errors

• set_region_error(v: errorpb::Error)
• set_kv_error(v: kvrpcpb::KeyError)
• set_cluster_id_error(v: ClusterIdError)

Getting Errors

• get_region_error() -> &errorpb::Error
• get_kv_error() -> &kvrpcpb::KeyError
• get_cluster_id_error() -> &ClusterIdError
• has_region_error() -> bool
• has_kv_error() -> bool
• has_cluster_id_error() -> bool

Mutable Access

If the specific error type is not already present, these methods will initialize it with a default value before returning a mutable reference:

• mut_region_error() -> &mut errorpb::Error
• mut_kv_error() -> &mut kvrpcpb::KeyError
• mut_cluster_id_error() -> &mut ClusterIdError