DbfDataReader

repository·main·Indexed 40 days ago

https://github.com/yellowfeather/dbfdatareader

A high-performance .NET library for reading dBase, xBase, Clipper, and FoxPro (.dbf) files. It features full ADO.NET DbDataReader compatibility, support for compound index (.cdx) files, and a SQL-like query engine via DbfDbConnection. The library supports LINQ-style typed queries, Dapper-compatible mapping, and high-performance bulk imports to SQL Server using SqlBulkCopy. It targets net10.0 and netstandard2.1.

Tokens
4.6K
Snippets
17
Records
17
Agent score
100%

What's inside DbfDataReader

  1. Search Visual FoxPro compound indexes (.cdx)

    main

    You can use CdxFile to open .cdx files and search for specific keys. The search results provide RecordIndex values (one-based) which can be converted to zero-based indices for use with DbfTable.Seek or DbfTable.Read.

    using DbfDataReader.Cdx;
    
    var dbfPath = "path/file.dbf";
    var cdxPath = "path/file.cdx";
    
    using (var dbfTable = new DbfTable(dbfPath))
    using (var cdxFile = new CdxFile(cdxPath, dbfTable.CurrentEncoding))
    {
        var tagNames = cdxFile.TagNames;             // the named indexes ("tags") in the file
    
        var index = cdxFile.GetIndex("CONTACT_ID");  // one tag; index.KeyExpression describes the key
        var dbfRecord = new DbfRecord(dbfTable);
    
        foreach (var entry in index.Search("C0000000042"))
        {
            dbfTable.Seek(entry.RecordIndex);
            dbfTable.Read(dbfRecord);
            // dbfRecord now holds the matching row
        }
    }
  2. Automatic index usage and ExplainPlan

    main

    The library automatically uses sidecar compound indexes (.cdx files) when available. Equality, range, BETWEEN, and prefix LIKE predicates on supported columns (character, integer, numeric, double, date) will trigger an index seek. ORDER BY clauses matching an index tag will read in index order.

    Important Notes:

    • The planner is conservative: it falls back to a full table scan for descending keys, expression keys, unsupported types (datetime, currency), or non-ASCII character searches.
    • You can force a full scan by setting UseIndexes=false in the connection string or calling .WithoutIndexes() on the builder.
    • Use DbfDbCommand.ExplainPlan() or DbfQuery<T>.ExplainPlan() to inspect the execution path.
    var command = (DbfDbCommand)dbConnection.CreateCommand();
    command.CommandText = "select * from setup.dbf where KEY_NAME = 'CONTACTS'";
    Console.WriteLine(command.ExplainPlan()); // Output example: index seek (=) on tag 'KEY_NAME'
  3. Run automatic index use benchmarks

    main

    To benchmark the performance of automatic index usage (comparing index seeks against full scans for various query types like character seeks, equality seeks, and range scans), run the following command. This uses a generated 50,000-row table with a compound index on ID and CODE.

    dotnet run -c Release --project test/DbfDataReader.Benchmarks -- --filter "*IndexQueryBenchmarks*"
  4. Run cross-library performance benchmarks

    main

    To compare DbfDataReader performance against other libraries (like Sylvan, NDbf, or DbfData) for reading every field of every record, use the following command.

    Note: You must set the version via the DbfDataReaderVersion environment variable. Using the -p:DbfDataReaderVersion MSBuild property will not work because it does not reach the BenchmarkDotNet generated child build. You can verify the version being measured by checking the // DbfDataReader under benchmark: line in the benchmark log.

    DbfDataReaderVersion=1.1.0 dotnet run -c Release --project test/DbfDataReader.Benchmarks -- --filter "*DbfDataReaderBenchmarks*" --job short
  5. Query a folder of DBF files using DbfDbConnection

    main

    You can treat a folder of DBF files as a database by using DbfDbConnection. This allows you to execute SQL commands against the files within a specified directory. The connection string must include the Folder option.

    To use it, set the ConnectionString with the Folder path, call .Open(), and then create a command to execute SQL text.

    var dbConnection = new DbfDbConnection(string.Empty, string.Empty);
    dbConnection.ConnectionString = $"Folder=./test/fixtures;SkipDeletedRecords=false";
    dbConnection.Open();
    
    var dbCommand = dbConnection.CreateCommand();
    dbCommand.CommandText = "select * from dbase_03.dbf;";
    
    var reader = await dbCommand.ExecuteReaderAsync();
    while (await reader.ReadAsync())
    {
        var valueCol1 = reader.GetString(0);
        var valueCol11 = reader.GetDecimal(10);
    }
  6. Execute scalar queries with DbfDbCommand

    main

    To execute a query that returns a single value (like a count or a specific column value), cast the created command to DbfDbCommand and use ExecuteScalar().

    var command = (DbfDbCommand)dbConnection.CreateCommand();
    command.CommandText = "select Point_ID from dbase_03.dbf where Point_ID = @id";
    command.Parameters.AddWithValue("@id", "A1");
    var value = command.ExecuteScalar();
  7. Bulk copy DBF data to SQL Server

    main

    Since DbfDataReader implements DbDataReader, you can pass it directly to SqlBulkCopy.WriteToServer to perform high-performance imports into MS SQL Server.

    var options = new DbfDataReaderOptions
    {
        SkipDeletedRecords = true
    };
    
    var dbfPath = "path/file.dbf";
    using (var dbfDataReader = new DbfDataReader(dbfPath, options))
    {
        using (var bulkCopy = new SqlBulkCopy(connection))
        {
            bulkCopy.DestinationTableName = "DestinationTableName";
    
            try
            {
                bulkCopy.WriteToServer(dbfDataReader);
            }
            catch (Exception ex)
            {
                Console.WriteLine($"Error importing: dbf file: '{dbfPath}', exception: {ex.Message}");
            }
        }
    }
  8. Iterate over rows using DbfTable and DbfRecord

    main

    To iterate through rows manually, use DbfTable.Read(DbfRecord). You can check DbfRecord.IsDeleted to decide whether to skip records marked as deleted.

    var skipDeleted = true;
    
    var dbfPath = "path/file.dbf";
    using (var dbfTable = new DbfTable(dbfPath, Encoding.UTF8))
    {
        var dbfRecord = new DbfRecord(dbfTable);
    
        while (dbfTable.Read(dbfRecord))
        {
            if (skipDeleted && dbfRecord.IsDeleted)
            {
                continue;
            }
    
            foreach (var dbfValue in dbfRecord.Values)
            {
                var stringValue = dbfValue.ToString();
                var obj = dbfValue.GetValue();
            }
        }
    }
  9. Use DbfDataReader for ADO.NET compatibility

    main

    The DbfDataReader class implements DbDataReader, allowing you to use standard ADO.NET methods like GetString, GetDecimal, GetDateTime, and GetInt32. You can configure it using DbfDataReaderOptions (e.g., SkipDeletedRecords).

    var options = new DbfDataReaderOptions
    {
        SkipDeletedRecords = true
    };
    
    var dbfPath = "path/file.dbf";
    using (var dbfDataReader = new DbfDataReader(dbfPath, options))
    {
        while (dbfDataReader.Read())
        {
            var valueCol1 = dbfDataReader.GetString(0);
            var valueCol2 = dbfDataReader.GetDecimal(1);
            var valueCol3 = dbfDataReader.GetDateTime(2);
            var valueCol4 = dbfDataReader.GetInt32(3);
        }
    }
  10. Perform typed queries with Query<T>

    main

    Use DbfTable.Query<T>() to map DBF rows directly to C# objects. The query engine translates LINQ-style lambdas (Where, OrderBy, Take) into optimized operations. Properties in your class must match column names (exact or case-insensitive).

    Supported Where operations:

    • Comparisons (==, !=, >=, etc.)
    • Logical operators (&&, ||, !)
    • Null checks (== null, != null)
    • String operations (StartsWith, EndsWith, Contains)
    • Collection membership (collection.Contains(p.Column))

    Note: Deleted records are skipped by default unless .IncludeDeleted() is called. Unsupported translations throw NotSupportedException.

    public class GpsPoint
    {
        public string Point_ID { get; set; }
        public decimal? Max_PDOP { get; set; }
        public DateTime? Date_Visit { get; set; }
    }
    
    using (var dbfTable = new DbfTable("path/dbase_03.dbf"))
    {
        var points = dbfTable.Query<GpsPoint>()
            .Where(p => p.Max_PDOP >= 3.5m && p.Point_ID.StartsWith("A"))
            .OrderByDescending(p => p.Date_Visit)
            .Take(10)
            .ToList(); // also foreach, First/FirstOrDefault, Count, ToListAsync, AsAsyncEnumerable
    }
  11. Inspect DBF table headers and columns with DbfTable

    main

    Use DbfTable to open a .dbf file and inspect its metadata, such as the version description, record count, and column definitions (name, type, length, and decimal count).

    var dbfPath = "path/file.dbf";
    using (var dbfTable = new DbfTable(dbfPath, Encoding.UTF8))
    {
        var header = dbfTable.Header;
    
        var versionDescription = header.VersionDescription;
        var hasMemo = dbfTable.Memo != null;
        var recordCount = header.RecordCount;
    
        foreach (var dbfColumn in dbfTable.Columns)
        {
            var name = dbfColumn.ColumnName;
            var columnType = dbfColumn.ColumnType;
            var length = dbfColumn.Length;
            var decimalCount = dbfColumn.DecimalCount;
        }
    }