gotgt SCSI Target Framework

While the project has high coverage for critical paths (Read/Write/Inquiry/Login/Logout), developers should be aware of the following implementation details and limitations:

Implemented Features

• COMPARE_AND_WRITE (0x89): SCSI atomic compare-and-write command.
• SNACK PDU: iSCSI error recovery mechanism (Data ACK, Status ACK, R2T retransmission).
• Async PDU: Asynchronous message notification mechanism.

Known Limitations / Unimplemented

• Multi-connection (MC/S): Not yet implemented (low priority).
• Error Recovery Level > 0: Requires complex configuration and is not fully tested.
• Header/Data Digest: Only partially tested.
• Large LUNs (>255): Requires special configuration and is not currently tested.
• FORMAT_UNIT: Implemented but requires special configuration for testing.

Gotgt uses sync.Pool for ISCSICommand objects and buffers to reduce GC pressure. For high-load production environments, you can pre-warm these pools during daemon initialization to avoid initial allocation spikes.

Example: Pre-allocating the command pool

func init() {
    // Pre-allocate 1000 commands into the pool
    for i := 0; i < 1000; i++ {
        cmd := &ISCSICommand{}
        commandPool.Put(cmd)
    }
}

To ensure the stability of the protocol stack and verify that optimizations haven't introduced concurrency issues, run the following commands:

Run race detector tests:

go test -race ./...

Run static analysis (vet):

go vet ./...

NUMA Tuning

1. Determine Topology: Use numactl --hardware or lscpu | grep NUMA to identify nodes.
2. Align Resources: Ensure network interfaces and storage devices are on the same NUMA node as the iSCSI process.
3. Buffer Sizing: Adjust numaBufferPoolSize (default 1024) and numaBufferSize (default 256KB) based on expected concurrent I/O.

io_uring Tuning

1. Queue Depth: Adjust ioUringQueueDepth (typical range: 128-4096). Higher values increase throughput but may increase latency.
2. I/O Size: Match application I/O size to the backend for efficiency. Use O_DIRECT to bypass the page cache.
3. System Limits: Ensure ulimit -n (nofile) is sufficiently high (e.g., 1048576) in /etc/security/limits.conf.

Since there are currently no release binaries, you must build gotgt from source. Ensure you have go and automake installed on your system. The repository should be cloned into your $GOPATH.

$ git clone https://github.com/gostor/gotgt
$ cd gotgt
$ make

To verify the compatibility and test coverage of the libiscsi integration, run the provided verification script from the repository root.

./test/verify_libiscsi_compat.sh

When building iSCSI protocol responses (found in pkg/port/iscsit/cmd.go, login.go, and logout.go), follow these patterns to reduce latency and allocations:

1. Pre-allocate exact buffer sizes: Avoid using append which causes re-allocations. Calculate the total required size (including padding) upfront.
2. Direct buffer writing: Instead of creating a temporary slice with Marshal and then using copy, use MarshalUint32To to write directly into the target buffer at the correct offset.
3. Bitwise padding calculation: Use bitwise operations for alignment, e.g., dl := (m.DataLen + 3) &^ 3.

// Optimized pattern: Single allocation and direct writes
buf := make([]byte, 48+rawDataLen+padding)
util.MarshalUint32To(buf[16:], m.TaskTag)
copy(buf[48:], m.RawData)

Use the following tools to validate performance improvements:

fio (Flexible I/O Tester) for disk I/O:

fio --name=iscsi-test --ioengine=libaio --iodepth=32 \
    --rw=randread --bs=4k --direct=1 --size=1G \
    --filename=/dev/sdX

iperf3 for network bandwidth:

iperf3 -c <target-ip> -p 3260

To optimize performance for large volumes of small I/O operations, adjust the command queue depth and outstanding R2T (Ready To Transfer) limits in your configuration.

Configuration Keys:

• MaxQueueCommand: Maximum number of commands in the queue.
• MaxOutstandingR2T: Maximum number of outstanding R2T requests.

Example Configuration:

{
  "MaxQueueCommand": 64,
  "MaxOutstandingR2T": 4
}

On Linux systems (kernel 5.1+), you can use io_uring for high-performance asynchronous I/O. This reduces system call overhead and improves performance for high queue depth workloads.

{
    "performance": {
        "enableIoUring": true,
        "ioUringQueueDepth": 4096
    },
    "storages": [
        {
            "deviceID": 1000,
            "path": "/var/tmp/disk.img",
            "online": true,
            "backendType": "iouring",
            "ioUringQueueDepth": 4096
        }
    ]
}

Enable NUMA optimizations in your configuration file using the performance block. Key settings include enableNUMA, numaBufferPoolSize (buffers per node), and numaBufferSize (size of each buffer in bytes).

{
  "performance": {
    "enableNUMA": true,
    "numaBufferPoolSize": 1024,
    "numaBufferSize": 262144
  }
}