repository·master·Indexed 23 days ago
https://github.com/valyala/bytebufferpoolA high-performance byte buffer pool implementation for Go designed to minimize memory waste and maximize speed. It provides a ByteBuffer type that implements io.ReaderFrom, io.WriterTo, and io.Writer, along with a Pool mechanism that automatically calibrates default and maximum buffer sizes based on workload patterns to reduce memory allocations and GC overhead.
bytebufferpool is a high-performance implementation of a pool of byte buffers designed for Go applications. Its primary goal is to provide efficient buffer reuse while including anti-memory-waste protection.
Note that the pool may waste a limited amount of memory due to fragmentation; this amount is approximately equal to the maximum total size of the byte buffers currently in concurrent use.
ByteBuffer type provides a byte buffer designed for append-like workloads, helping to minimize memory allocations. It is intended to be used with functions that append data to a []byte slice. You should obtain an empty buffer using Get (from the package level) rather than manual instantiation to leverage the pool.The bytebufferpool package provides a global defaultPool for easy access to byte buffer pooling without manual instantiation. Use Get() to acquire a *ByteBuffer and Put() to return it to the pool. This pattern reduces memory allocations and GC overhead.
Important: Once a *ByteBuffer is returned to the pool via Put(), you must not access its underlying byte slice (B) again, as this will cause data races.
ByteBuffer struct contains a single exported field B []byte. It implements several standard Go interfaces including io.ReaderFrom, io.WriterTo, and io.Writer.If you need to manage distinct types of byte buffers separately to reduce memory waste, you can instantiate your own Pool. A custom pool allows you to control the lifecycle of buffers specific to a certain workload.
Methods:
Get(): Returns an empty *ByteBuffer. If the pool is empty, it allocates a new one with a capacity based on the pool's current defaultSize.Put(b *ByteBuffer): Returns the buffer to the pool. The pool automatically calibrates its defaultSize and maxSize based on the sizes of buffers being returned to optimize for your specific usage patterns.To reuse a ByteBuffer without reallocating, use these methods:
Reset(): Makes the buffer empty by setting the length to 0 (retains capacity).Set(p []byte): Sets the buffer content to the provided slice p (retains capacity).SetString(s string): Sets the buffer content to the provided string s (retains capacity).The ByteBuffer provides several ways to append data, making it compatible with bytes.Buffer patterns:
Write(p []byte) (int, error): Appends the byte slice p to the buffer. Implements io.Writer.WriteString(s string) (int, error): Appends the string s to the buffer.WriteByte(c byte) error: Appends a single byte c to the buffer.To retrieve the accumulated data, use:
Bytes() []byte: Returns the underlying byte slice B.String() string: Returns the string representation of the buffer.ReadFrom(r io.Reader) method implements io.ReaderFrom. It appends all data read from the provided reader r into the buffer. It handles buffer expansion internally.Len() int method returns the current size of the byte buffer.WriteTo(w io.Writer) (int64, error) method implements io.WriterTo. It writes all accumulated bytes in the buffer to the provided writer w.The Pool struct manages a collection of *ByteBuffer objects using an internal sync.Pool. It features an automatic calibration mechanism that tracks the frequency of different buffer sizes being used.
Over time, the pool adjusts its defaultSize (the capacity of newly allocated buffers) and maxSize (the threshold below which buffers are kept in the pool) to match the actual workload, minimizing both allocation frequency and memory waste.