repository·master·Indexed 12 days ago
https://github.com/tencent/ncnnA high-performance neural network inference framework optimized for mobile, embedded, and desktop deployment. It features no third-party runtime dependencies and supports CPU and Vulkan GPU backends. Includes the benchncnn tool for measuring inference performance and benchncnn_llm for benchmarking Large Language Models.
ncnn is a high-performance neural network inference framework with the following characteristics:
PNNX offers several advantages for model deployment and interoperability:
.param file that is easy to read and edit..bin format stored within a zip archive.torch functions, and miscellaneous modules as single operators rather than breaking them down into many small pieces.Element packing is a technique used to store multiple short-sized values as a single long-sized value. This is designed to map efficiently to SIMD (Single Instruction, Multiple Data) registers, which use wide registers to process multiple values simultaneously.
When using elempack > 1, the ncnn::Mat structure treats the wide-sized value as a single element. Consequently, the logical width or dimension of the Mat is reduced by the elempack factor.
Example:
If you want to store 40 float values (where elemsize is 4):
elempack = 1: Mat width is 40.elempack = 4: Mat width is 10.Common Packing Mappings:
| Type | elemsize | elempack |
|---|---|---|
double | 8 | 1 |
float | 4 | 1 |
int | 4 | 1 |
short | 2 | 1 |
signed char | 1 | 1 |
ARM NEON Mappings:
| Type | elemsize | elempack |
|---|---|---|
float64x2_t | 16 | 2 |
float32x4_t | 16 | 4 |
int32x4_t | 16 | 4 |
float16x4_t | 8 | 4 |
int8x8_t | 8 | 8 |
On devices with unified memory, you can access GPU memory directly from the CPU by using the .mapped() method on an ncnn::VkMat. This allows you to use the pointer returned by mapped().data directly without explicit copies.
ncnn::VkMat blob_gpu;
ncnn::Mat mapped = blob_gpu.mapped();
// use mapped.data directlyIn ncnn, the Reshape layer can use an expression (indicated by the prefix 6=) to define dynamic shapes or subscript values based on input shapes. This is more efficient than using multiple arithmetic operators because it reduces model complexity and avoids the overhead of kernel calls for simple single-digit operations by performing them directly on the CPU.
When using the pnnx tool, pnnx.Expression and Tensor.reshape/Tensor.view operators are automatically fused into a single ncnn Reshape layer with an expression string.
Example Conversion:
Python (pnnx):
shape = [(B.size(0) + 2), (A.size(1) * 2), -1]
out = A.reshape(*shape)ncnn.param:
Reshape reshape 2 1 A B out 6="-1,*(0h,2),+(1c,2)"Reshape reshape 2 1 A B out 6="-1,*(0h,2),+(1c,2)"Creating an ImportAndroidHardwareBufferPipeline is expensive (~24 ms median on Adreno 830). Since the pipeline parameters (sampler, rotation, target size) are usually stable for a camera session, you should cache the allocator and pipeline using the AHardwareBuffer* as a key.
Note: VkAndroidHardwareBufferImageAllocator does not take a reference on the AHB. If you cache the allocator, you must call AHardwareBuffer_acquire(ahb) to ensure the buffer remains valid, and AHardwareBuffer_release(ahb) when evicting the cache.
struct CacheEntry {
ncnn::VkAndroidHardwareBufferImageAllocator* alloc;
ncnn::ImportAndroidHardwareBufferPipeline* pipe;
ncnn::VkImageMat src;
};
static std::unordered_map<AHardwareBuffer*, CacheEntry> cache; // camera-thread only
auto it = cache.find(ahb);
if (it == cache.end()) {
AHardwareBuffer_acquire(ahb);
auto* a = new ncnn::VkAndroidHardwareBufferImageAllocator(vkdev, ahb);
auto s = ncnn::VkImageMat::from_android_hardware_buffer(a);
auto* p = new ncnn::ImportAndroidHardwareBufferPipeline(vkdev);
p->create(a, 1, 1, width, height, opt);
cache[ahb] = { a, p, std::move(s) };
it = cache.find(ahb);
}
const CacheEntry& e = it->second;
cmd.record_import_android_hardware_buffer(e.pipe, e.src, dst);The ncnn::Layer base class provides four interfaces. You must implement the one marked as must for your specific combination of one_blob_only and support_inplace. Implementing the optional version can improve performance by avoiding deep copies.
one_blob_only | support_inplace | 1 (Multi-blob forward) | 2 (Single-blob forward) | 3 (In-place multi-blob) | 4 (In-place single-blob) |
|---|---|---|---|---|---|
false | false | must | |||
false | true | optional | must | ||
true | false | must | |||
true | true | optional | must |
Interface Definitions:
virtual int forward(const std::vector<Mat>& bottom_blobs, std::vector<Mat>& top_blobs, const Option& opt) const; (Multi-input/output)virtual int forward(const Mat& bottom_blob, Mat& top_blob, const Option& opt) const; (Single-input/output)virtual int forward_inplace(std::vector<Mat>& bottom_top_blobs, const Option& opt) const; (Multi-input/output, in-place)virtual int forward_inplace(Mat& bottom_top_blob, const Option& opt) const; (Single-input/output, in-place)By default, ncnn keeps blobs (intermediate results) in memory. For most deep networks, you only need the final result or specific branch results. Enabling Light Mode allows ncnn to automatically reclaim memory for blobs that are no longer needed after a layer's computation is complete. This significantly reduces the memory footprint during inference.
Example behavior:
In a network A -> B -> C, when requesting result C in Light Mode:
A's result is reclaimed when B starts computing.B's result is reclaimed when C starts computing.C is retained in memory.ncnn includes a built-in Vulkan loader called SimpleVK (enabled via the NCNN_SIMPLEVK CMake option when NCNN_VULKAN is on).
SimpleVK allows you to use Vulkan without requiring the full Vulkan SDK on the development machine or the target system. It can dynamically load the Vulkan runtime or graphics drivers at runtime, making it easier to distribute applications without explicit libvulkan linkage.
For most users, manual management of the Vulkan instance is unnecessary; simply enabling the Vulkan compute option in the ncnn::Net configuration is sufficient.
ncnn::Net net;
net.opt.use_vulkan_compute = true;
net.load_param("model.param");
net.load_param("model.bin");PNNX can resolve all tensor shapes in a model graph and constantify common expressions when shapes are known. This is an optional process enabled by providing the inputshape command line option.
Providing inputshape allows PNNX to transform dynamic operations (like view or reshape based on input size) into static, optimized operations.
pnnx shufflenet_v2_x1_0.pt inputshape=[1,3,224,224]