For more robust training pipelines, implement a custom torch.utils.data.Dataset. This allows you to leverage torch.utils.data.DataLoader, which provides features like automatic batching and data shuffling.
To implement a custom dataset:
- Inherit from
torch.utils.data.Dataset. - Implement the constructor to load your data (e.g., via
torch.Tensor.load()). - Override
GetTensor(long index) to return a Dictionary<string, torch.Tensor> containing the sample and its label. - Override the
Count property to return the number of items.
Once implemented, pass the dataset to a DataLoader with desired batch_size and shuffle settings.
class SyntheticDataset : torch.utils.data.Dataset {
public SyntheticDataset(string fileName) {
_data = torch.Tensor.load(fileName + ".x");
_labels = torch.Tensor.load(fileName + ".y");
}
public override Dictionary<string, torch.Tensor> GetTensor(long index) {
return new Dictionary<string, torch.Tensor> {
{ "data", _data[(int)index] },
{ "label", _labels[(int)index] }
};
}
public override long Count => _data.shape[0];
private torch.Tensor _data;
private torch.Tensor _labels;
}
// Usage:
var training_data = new SyntheticDataset("train.dat");
var train = new torch.utils.data.DataLoader(training_data, 200, shuffle: true);
