DOCSEARCHbyAlgolia
Submit Docs

PingCAP Talent Plan

repository·master·Indexed 27 days ago

https://github.com/pingcap/talent-plan

A Rust-based educational project for learning distributed systems. The curriculum includes the 'Distributed Systems in Rust' course, covering the Raft consensus algorithm (leader election, log replication, and snapshotting), the Percolator distributed transaction protocol (TSO and Storage Servers), and the implementation of a fault-tolerant key-value storage service (KvRaft) and a basic key-value store (kvs).

Tokens
34.1K
Snippets
67
Records
268
Agent score
94%

What's inside pingcap-talent-plan

  1. Overview of TP 101: Introduction to Open Source Software

    master

    TP 101 is an introductory course designed to teach the key concepts of open source software. It covers the history of open source, various licensing models, and the role of open source organizations and foundations.

    Note: This course is currently in an alpha state. Users are encouraged to report bugs or suggest changes via the project's Contributing Guide.

  2. Overview of Project: Asynchrony

    master

    Project: Asynchrony is a task to create a multi-threaded, persistent key/value store server and client using asynchronous networking over a custom protocol.

    Key Objectives:

    • Implement asynchronous networking using the tokio runtime.
    • Use Rust futures (specifically version 0.1) to handle asynchronous tasks.
    • Experiment with different future type definitions: boxed futures (Box<Future>), explicit future types, and anonymous future types (impl Trait).
    • Implement a dual-layer thread pool architecture: one layer for asynchronous networking (via tokio) and a second layer for synchronous file I/O via a custom ThreadPool.
    • Convert the KvsClient and KvsEngine to present futures-based APIs.

  3. Overview of PNA Rust Projects

    master

    The course builds a networked key-value database through several stages:

    1. Project 1: The Rust toolbox: Create an in-memory key/value store. Focuses on cargo tools, clap for CLI parsing, and basic data types.
    2. Project 2: Log-structured file I/O: Create a persistent key/value store. Focuses on serde for serialization, log-structured file I/O (Bitcask algorithm), and error handling.
    3. Project 3: Synchronous client-server networking: Create a single-threaded client-server application using a custom protocol over std::net. Focuses on traits for pluggable backends and logging.
    4. Project 4: Concurrency and parallelism: Upgrade the server to be multithreaded. Focuses on thread pools, channels, locks, and lock-free data structures.
    5. Project 5: Asynchronous programming in Rust: (Coming soon) Focuses on asynchronous I/O.

  4. Overview of Distributed Systems in Rust course

    master
    The 'Distributed Systems in Rust' course is a training program designed to teach Rust programmers how to build reliable distributed systems and implement distributed transactions. The course covers the Raft consensus algorithm (including building a fault-tolerant key-value storage service) and the Percolator transaction model. Upon completion, participants should be able to implement a basic, fault-tolerant key-value storage service with transactions in Rust.

  5. Overview of TP 103: Build A Welcoming Community

    master

    TP 103 is a course designed for open source project owners and enthusiasts to learn how to build and maintain a welcoming community. The course covers roles within an open source organism, community governance models, strategies for community vibrancy, and leadership principles, using TiDB as a practical case study.

    Note: This course is currently in an alpha state. Users are encouraged to report bugs or suggest changes via the project's Contributing Guide.

  6. Overview of PNA Rust Project 4: Concurrency and parallelism

    master
    This project involves creating a multi-threaded, persistent key/value store server (kvs-server) and client (kvs-client) using synchronous networking over a custom protocol. The goal is to implement a thread pool, use channels for cross-thread communication, share data structures with locks, and perform benchmarking to compare single-threaded vs. multithreaded performance.

  7. Overview of PNA Rust Project 2: Log-structured file I/O

    master
    This project involves building a persistent key/value store that can be accessed via a command-line interface. The implementation uses a log-structured approach (similar to Bitcask) where write operations are appended to an on-disk log, and an in-memory index maps keys to their locations (log pointers) in that log. Key features include robust error handling, serialization using serde, and periodic log compaction to reclaim disk space.

  8. Overview of PNA Rust Project 3

    master
    This project involves creating a single-threaded, persistent key/value store consisting of a server (kvs-server) and a client (kvs-client). The components communicate over a custom synchronous networking protocol using std::net. The project focuses on implementing a custom storage engine, using traits for pluggable backends (including sled), and implementing robust logging and error handling across network boundaries.

  9. Overview of Practical Networked Applications in Rust

    master

    Practical Networked Applications in Rust is a training course designed to teach novice Rust programmers how to build high-performance, reliable, and concurrent systems software. Through a series of projects, students build a networked, multithreaded, and asynchronous key-value database.

    Key subjects covered include:

    • Structuring and maintaining Rust programs
    • Using tools like clippy and rustfmt
    • Error handling best practices
    • Serialization with serde
    • Log-structured storage (inspired by bitcask)
    • Network programming with std and tokio
    • Benchmarking with criterion
    • Parallel programming with crossbeam
    • Asynchronous programming with futures

  10. Requirements and Evaluation Criteria

    master

    Your implementation is evaluated based on the following criteria:

    • Correctness (30%): Must pass all test cases (including JoinTest).
    • Performance (20%): Must perform better than the baseline implementation in join_example.go:JoinExample.
    • Documentation (35%): Provide a document describing your approach and your performance optimization process using pprof.
    • Code Quality (15%): Maintain good code style.

  11. Explore Distributed Systems in Go curriculum

    master

    The talent-plan repository contains a series of weekly labs focused on distributed systems implemented in Go. You can explore the following modules:

    • Week 1: Merge Sort - Implementation of the merge sort algorithm.
    • Week 2: Map Reduce - Implementation of the MapReduce framework.
    • Week 4: Parallel Join - Implementation of parallel join operations.

  12. Access Talent Plan 1.0 Learning Paths

    master

    Talent Plan 1.0 offers two distinct learning paths based on your area of interest:

    • TiDB Direction: Focused on the SQL layer of distributed relational databases. See 1.0-lp-tidb.md for details.
    • TiKV Direction: Focused on the storage layer of distributed Key-value databases. See 1.0-lp-tikv.md for details.