DOCSEARCHbyAlgolia
Submit Docs

SharpToken

repository·main·Indexed 42 days ago

https://github.com/dmitry-brazhenko/sharptoken

A high-performance C# port of OpenAI's tiktoken library for encoding, decoding, and counting tokens for GPT-based models. Optimized for .NET 8, .NET 6, and .NET Standard 2.0, it provides efficient APIs including Encode, Decode, and CountTokens. Supports various encodings such as cl100k_base, o200k_base, and Claude models.

Tokens
2.9K
Snippets
7
Records
10
Agent score
51%

What's inside SharpToken

  1. Performance benchmarks for SharpToken

    main

    SharpToken provides performance benchmarks comparing its methods (Encode, CountTokens, and CountTokens_LargeInput) across different .NET runtimes: .NET 8.0, .NET 6.0, and .NET Framework 4.7.1.

    Key performance observations:

    • .NET 8.0 Optimization: The library is highly optimized for .NET 8.0. For example, CountTokens_LargeInput on .NET 8.0 allocates only 75 B and is significantly faster than on older runtimes.
    • Allocation Efficiency: Recent improvements (v1.2.17+) have focused on reducing minor allocations, particularly for the CountTokens method in .NET 8.0, which can achieve near-zero allocation (approx. 4.06 KB).
    • Runtime Comparison: .NET 8.0 consistently shows the lowest Mean execution time and lowest Allocated memory across all tested methods compared to .NET 6.0 and .NET Framework 4.7.1.

  2. Performance characteristics of SharpToken

    main

    SharpToken is optimized for high performance and low memory allocation, particularly on .NET 8.0, where it utilizes modern multibyte CPU instructions. It is designed to be faster and more memory-efficient than alternatives like TiktokenSharp and TokenizerLib.

    Core Methods and Performance

    • Encode: Converts text to tokens.
    • Decode: Converts tokens back to text.
    • CountTokens: A high-performance API specifically for counting tokens in text without full encoding/decoding.

    Benchmarks Summary (.NET 8.0)

    MethodInput SizeMean TimeAllocated Memory
    EncodeSmall (453 B)~22.65 us696 B
    EncodeLarge (51 KB)~4.54 ms~155 KB
    DecodeSmall (453 B)~1.62 us~2.3 KB
    DecodeLarge (51 KB)~454.57 us~287 KB
    CountTokensSmall (453 B)~22.01 us184 B
    CountTokensLarge (51 KB)~4.23 ms~195 B

  3. Understand SharpToken performance and runtime optimizations

    main

    SharpToken is optimized for different .NET runtimes. Performance and memory allocation characteristics vary significantly between .NET 8.0, .NET 6.0, and .NET Framework 4.7.1.

    Key takeaways for developers:

    • .NET 8.0 provides the best performance and lowest memory allocations. It benefits from specific optimizations like ReadOnlySpan<char> support, compiled regex, and a fast MultiBytePairEncoder with near-zero allocations.
    • .NET 6.0 and .NET Framework 4.7.1 are supported, but generally exhibit higher latency and higher memory allocations compared to .NET 8.0.
    • Memory Management: In .NET 8.0, many operations (like Encode) produce significantly less garbage (Gen0/Gen1) and lower total allocated bytes compared to older runtimes.

  4. Install SharpToken via NuGet or .NET CLI

    main

    You can install SharpToken using the NuGet package manager or the .NET CLI. It is compatible with .NET 6, .NET 8, and .NET Standard 2.0.

    # Using NuGet Package Manager
Install-Package SharpToken

# Using .NET CLI
dotnet add package SharpToken

  5. Run the SharpToken Benchmark

    main

    To run the encoding benchmark, which tests performance against samples in GptEncoderTestSamples.txt, you must use a release build of the benchmark executable.

    Navigate to the following directory and execute SharpToken.Benchmark.exe: ..\SharpToken\SharpToken.Benchmark\bin\Release\net8.0\SharpToken.Benchmark.exe

    # Navigate to the release directory and run the executable
# Path: ..\SharpToken\SharpToken.Benchmark\bin\Release\net8.0\SharpToken.Benchmark.exe
.\SharpToken.Benchmark.exe

  6. Basic usage of encoding and decoding tokens

    main

    To use SharpToken, import SharpToken and create an instance of GptEncoding using either an encoding name or a model name. You can then use Encode to convert text to integers, Decode to convert integers back to text, and CountTokens for high-performance token counting (useful for RAG chunking or checking prompt limits).

    using SharpToken;

// Get encoding by encoding name
var encoding = GptEncoding.GetEncoding("cl100k_base");

// Or get encoding by model name
var encodingByModel = GptEncoding.GetEncodingForModel("gpt-4");

// Encode a string
var encoded = encoding.Encode("Hello, world!"); // Output: [9906, 11, 1917, 0]

// Decode tokens back to string
var decoded = encoding.Decode(encoded); // Output: "Hello, world!"

// High performance token counting
var count = encoding.CountTokens("Hello, world!"); // Output: 4

  7. Map model names to encodings using Model.GetEncodingNameForModel

    main

    SharpToken provides prefix matching to map specific model names to their corresponding encodings. Use Model.GetEncodingNameForModel(string modelName) to retrieve the encoding name. If the model name does not match a known prefix or direct name, an exception is thrown.

    // Returns "claude"
string encodingName = Model.GetEncodingNameForModel("claude-3.5-sonnet"); 

// Returns "cl100k_base"
string encodingName = Model.GetEncodingNameForModel("gpt-4-0314");

  8. Encode with custom allowed or disallowed special tokens

    main

    You can control how special tokens are handled during encoding by passing a HashSet<string> to the Encode method.

    • Allowed Tokens: Pass a HashSet<string> to Encode to specify which special tokens should be treated as valid tokens rather than plain text.
    • Disallowed Tokens: Pass a HashSet<string> to the disallowedSpecial parameter. If the input text contains any of these tokens, an ArgumentException will be thrown.
    // Custom Allowed Sets
const string encodingName = "cl100k_base";
const string inputText = "Some Text <|endofprompt|>";
var allowedSpecialTokens = new HashSet<string> { "<|endofprompt|>" };

var encoding = GptEncoding.GetEncoding(encodingName);
var encoded = encoding.Encode(inputText, allowedSpecialTokens);

// Custom Disallowed Sets
void TestAction()
{
    // This will throw ArgumentException if "Some" is in the input text
    encoding.Encode(inputText, disallowedSpecial: new HashSet<string> { "Some" });
}

  9. Supported encodings and models

    main

    SharpToken supports several base encodings and specific model mappings. You can instantiate an encoding directly using GptEncoding.GetEncoding(string encodingName).

    Supported Encodings:

    • r50k_base
    • p50k_base
    • p50k_edit
    • cl100k_base
    • o200k_base
    • o200k_harmony
    • claude (Uses Anthropic's official vocabulary with NFKC normalization; accurate for pre-Claude 3, approximate for Claude 3+)

    Claude Model Support: All claude-* model names (e.g., claude-3-opus, claude-3.5-sonnet, claude-3.7-sonnet) are supported via GetEncodingForModel.

    var r50kBaseEncoding = GptEncoding.GetEncoding("r50k_base");
var p50kBaseEncoding = GptEncoding.GetEncoding("p50k_base");
var p50kEditEncoding = GptEncoding.GetEncoding("p50k_edit");
var cl100kBaseEncoding = GptEncoding.GetEncoding("cl100k_base");
var o200kBaseEncoding = GptEncoding.GetEncoding("o200k_base");
var o200kHarmonyEncoding = GptEncoding.GetEncoding("o200k_harmony");
var claudeEncoding = GptEncoding.GetEncoding("claude");

  10. Compare Encode and CountTokens performance across runtimes

    main

    The following benchmark data (from version 1.2.17) compares the Encode and CountTokens methods. Note that these results represent allocations produced specifically within the benchmark method and include overhead from List.Add(), File.ReadAllLines(), and IEnumerable<string>.GetEnumerator().

    | Method      | Job                  | Runtime             | Mean       | Error    | StdDev   | Ratio | Gen0     | Gen1   | Allocated | Alloc Ratio |
|------------ |--------------------- |---------------------|-----------:|---------:|---------:|------:|---------:|--------:|-----------:|------------:|
| Encode      | .NET 6.0             | .NET 6.0            | 1,113.1 us |  4.59 us |  3.59 us |  0.75 |  64.4531 | 5.8594 | 537.97 KB |        0.83 |
| Encode      | .NET 8.0             | .NET 8.0            |   649.5 us |  6.04 us |  5.65 us |  0.44 |   5.8594 |      - |  51.48 KB |        0.08 |
| Encode      | .NET Framework 4.7.1 | .NET Framework 4.7.1 | 1,481.1 us | 13.32 us | 11.81 us |  1.00 | 105.4688 | 9.7656 | 652.02 KB |        1.00 |
|             |                      |                     |           |         |         |       |         |        |           |             |
| CountTokens | .NET 6.0             | .NET 6.0            | 1,038.1 us | 10.00 us |  9.36 us |  0.73 |  60.5469 | 1.9531 | 505.07 KB |        0.81 |
| CountTokens | .NET 8.0             | .NET 8.0            |   603.3 us |  1.81 us |  1.51 us |  0.42 |   1.9531 |      - |  22.46 KB |        0.04 |
| CountTokens | .NET Framework 4.7.1 | .NET Framework 4.7.1 | 1,429.1 us | 19.10 us | 17.86 us |  1.00 | 99.6094 | 3.9063 |  622.4 KB |        1.00 |