DOCSEARCHbyAlgolia
Submit Docs

ozline's Technical Blog

website·Indexed 30 days ago

https://ozline.icu/

A technical blog featuring deep dives into Go 1.24 and 1.26, Kubernetes infrastructure (client-go, apimachinery), distributed systems consistency models, and RocketMQ refactoring with RocksDB. Includes performance analysis notes from CMU 15-418 and personal reflections on software engineering and professional growth.

Tokens
14.9K
Snippets
28
Records
78
Agent score
43%

What's inside ozline Blog

  1. Overview of Go 1.24 sync.Map HashTrieMap implementation

    Starting with Go 1.24, sync.Map uses HashTrieMap as its default implementation. It is implemented as a hash prefix tree (Trie) where keys are hashed and the resulting hash bits are used to traverse the tree. The implementation uses a 16-child node design (4 bits per level) to balance search depth and memory usage.

  2. Understand Kubernetes architecture: Clusters, Nodes, and Pods

    Kubernetes organizes resources in a three-level hierarchy:

    1. Cluster: The highest-level deployment unit. It is a collection of machines that act as a single entity, abstracting multiple physical or virtual servers into one unified resource pool.
    2. Node: A single machine (physical or virtual) within a cluster. There are two types:
      • Master Node: Runs the control plane (API server, scheduler, controller manager) and manages cluster state and scheduling.
      • Worker Node: Runs the actual application containers and manages them via Kubelet and Kube-Proxy.
    3. Pod: The smallest deployable unit in Kubernetes. A Pod encapsulates one or more containers that share the same network namespace (IP address and ports), storage, and OS context. All containers in a Pod must reside on the same Node.

  3. Understand the relationship between CRDs and CRs

    In Kubernetes, Custom Resource Definitions (CRDs) and Custom Resources (CRs) allow users to extend the API with their own object types.

    • CRD (Custom Resource Definition): Acts like an interface. Applying a CRD registers a new API path in the Kubernetes API server, defining the resource's name, API group, version, and metadata. It tells Kubernetes that a new type of resource exists but does not define a specific instance of it.
    • CR (Custom Resource): Acts like an implementation of the CRD. A CR is a specific instance of the resource defined by the CRD, containing the spec (the desired state).

    Controllers monitor these CRs via the Informer mechanism and take action to make the cluster's actual state match the desired state defined in the CR's spec.

  4. Understand the role of the Kubernetes apimachinery library

    The apimachinery library serves as the foundation for Kubernetes development by providing generic, extensible, and reusable components. It defines common object structures (such as ObjectMeta and TypeMeta), client connection settings (like rest.Config), and resource handling interfaces (such as runtime.Object). It is designed to provide a unified standard that is used by client-go and custom controllers to ensure consistency across the Kubernetes ecosystem without implementing specific business logic.

  5. Use PersistentVolumeClaims (PVC) and ConfigMaps for storage

    Kubernetes abstracts storage so developers don't need to worry about the underlying implementation (e.g., NFS, Cloud Storage, vSphere). Storage is mounted into containers similarly to the Linux mount command.

    Key Storage Types:

    • PersistentVolumeClaim (PVC): Used for persistent data. The application inside the container can read and write to this storage, and changes persist even if the Pod restarts.
    • ConfigMap: Used for application configuration (typically key-value pairs). It is mounted as a volume into the container. Unlike PVCs, ConfigMaps are read-only for the application; changes must be made by an administrator in the backend.

  6. Address Remoting SDK compatibility issues in RocketMQ 5.0

    Legacy Remoting SDKs (versions prior to 4.9.5) cannot connect to RocketMQ 5.0 because they do not specify the BrokerName in their protocol. In the 5.0 architecture, clients connect to the Proxy rather than the Broker; without the BrokerName, the Proxy cannot determine which Broker to forward the request to. While version 4.9.5 and 5.x SDKs include this field, a significant portion of users on older SDKs (4.7, 4.8) remain incompatible and cannot use POP consumption.

  7. Distinguish Consensus Protocols from Consistency Models

    Consensus (e.g., Raft, Paxos) and Consistency are distinct concepts. A consensus protocol ensures that a group of nodes agrees on a single value or a sequence of logs, but it does not inherently define the consistency model for the application's data access. For example, Raft's log implementation provides sequential consistency via total order broadcast, but the overall system consistency depends on how the upper-layer application handles reads and writes.

  8. Implement scheduled message storage using RocksDB

    In the RocksDB-refactored RocketMQ storage engine, scheduled messages are managed by replacing the traditional time-wheel concept with a KV-based scanning approach. Messages are stored as keys where the timestamp is modified to allow for efficient scanning. When a message is scanned, its timestamp is updated to shift it forward until it is ready to be moved into the actual consumption queue. This model eliminates the need for fixed-length files for each time interval, preventing space waste for empty intervals. The lifecycle is simplified to: enqueue (write key) and dequeue (delete key).

  9. Understand the CAP Theorem trade-offs in distributed systems

    The CAP theorem states that a distributed system can only simultaneously provide two of the following three guarantees:

    • Consistency (C): Every read receives the most recent write or an error. Data is consistent across all nodes; any node read returns the correct value, and writes are synchronized across all nodes.
    • Availability (A): Every request receives a (non-error) response, without the guarantee that it contains the most recent write.
    • Partition Tolerance (P): The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes (e.g., node crashes, network failure).

    In practice, if a system chooses high availability and partition tolerance (AP), it must sacrifice strong consistency. If it chooses consistency and partition tolerance (CP), it must sacrifice availability during synchronization periods.

  10. RocketMQ 5.0 Architecture: Proxy and POP Consumption

    RocketMQ 5.0 introduces a Proxy component to achieve separation of storage and computation. While the Broker continues to handle storage (CommitLog and ConsumerQueue), the Proxy manages routing, filtering, distribution, and consumer pull requests. This enables more flexible AutoScaling and better resource utilization. Additionally, 5.0 introduces the POP consumption mode, which moves from queue-binding (where one queue is assigned to one client) to message-side load balancing, supported by a new gRPC-based SDK.

  11. Understand the limitations of RocketMQ's legacy queue-based storage

    RocketMQ's legacy storage model relies on file-system queues (FIFO) for various business functions, which introduces several bottlenecks:

    • Complexity: Implementing non-FIFO logic (like retries or scheduled tasks) requires constant dequeuing and re-queuing (merging), leading to high I/O overhead and complex code.
    • Scalability: Performance degrades significantly when the number of queues exceeds 100,000, and high file usage leads to file descriptor limits.
    • Efficiency: The FIFO nature forces data dependencies; for example, failed messages must be written back to the end of the queue, increasing latency and wasting storage space.
    • Indexing: The legacy index model (simulating a HashMap via files) suffers from high file copy overhead and hash collisions when the number of keys is large.

  12. Understand RocketMQ's traditional CommitLog and Queue storage architecture

    RocketMQ's traditional storage engine relies on a Write-Ahead-Log (WAL) implementation called CommitLog, which uses an Append-Only file system approach (similar to Redis AOF). Data is written sequentially (FIFO), and the system supports both synchronous writing (high reliability) and asynchronous writing (buffered for performance, but risk of data loss).

    Message queues are managed via offsets using batch sequential commits. Metadata (Topics, subscriptions, consumption offsets, and CheckPoints) is also stored on the file system. In high-load scenarios with thousands of Topics, this architecture can lead to performance degradation and OS file descriptor (FD) limits due to the massive number of files being written simultaneously.