dbfread

repository·master·Indexed 42 days ago

https://github.com/olemb/dbfread

A pure Python library for reading DBF files (dBase, Visual FoxPro, and FoxBase+) and converting them into native Python data types. It supports streaming records from disk for memory efficiency or loading them into memory via the DBF class. Features include support for memo files (.FPT, .DBT), custom field parsing by subclassing FieldParser, and utilities for exporting data to CSV, Pandas DataFrames, and SQL via the dbf2sqlite CLI tool.

Tokens
5.4K
Snippets
21
Records
26
Agent score
50%

What's inside dbfread

  1. Stream or load records into memory

    master

    By default, dbfread streams records directly from disk, meaning only one record is in memory at a time. This is memory-efficient for large files.

    If you need random access to records, you can load them into memory using the load=True argument. This populates the .records attribute with a list of records. You can also load records later by calling .load() on an existing DBF object. To free up memory, use .unload().

    If you only need a simple list of records without the table metadata, you can cast the DBF object to a list.

    from dbfread import DBF
    
    # 1. Streaming (Default): Memory efficient, one record at a time
    table = DBF('people.dbf')
    for record in table:
        pass
    
    # 2. Loading into memory: Allows random access via table.records
    table = DBF('people.dbf', load=True)
    print(table.records[1]['NAME'])
    
    # 3. Manual loading: Useful for inspecting headers before loading
    table = DBF('people.dbf')
    # ... do something with table header ...
    table.load()
    
    # 4. Unloading: Free memory
    table.unload()
    
    # 5. Quick list conversion
    records = list(DBF('people.dbf'))
  2. Manage record loading in memory

    master

    You can control whether records are streamed from disk or held in memory using the load() and unload() methods and the load argument during initialization.

    • Streaming (Default): When load=False, the records and deleted attributes are RecordIterator objects that stream data from the disk. This is memory-efficient for large files.
    • In-Memory: When load=True, or after calling .load(), the records and deleted attributes become lists. This allows for faster access but consumes more RAM.

    Use .unload() to clear records from memory and return to a streaming RecordIterator state.

    dbf = DBF('data.dbf', load=False)
    
    # Load everything into memory as lists
    dbf.load()
    print(len(dbf.records))
    
    # Unload to save memory and return to streaming mode
    dbf.unload()
  3. Access records via the .records attribute instead of list indexing

    master

    In version 2.0, the DBF class is no longer a subclass of list. While iteration and record counting (e.g., len(table)) still work as expected, direct list operations like indexing (e.g., table[1]) will no longer work on a DBF instance.

    To access specific records by index, use the .records attribute.

    # Old way (broken in 2.0+ if using DBF class)
    table = dbfread.read('people.dbf')
    print(table[1])
    
    # New way
    from dbfread import DBF
    table = DBF('people.dbf', load=True)
    print(table.records[1])
  4. Export DBF data to SQL using the `dataset` package

    master

    The dataset package can be used to easily move data from a DBF file into a modern database like SQLite. This method also handles schema creation automatically.

    import dataset
    from dbfread import DBF
    
    db = dataset.connect('sqlite:///example.sqlite')
    table = db['people']
    
    for record in DBF('example.dbf'):
        table.insert(record)
  5. Handle special characters in custom field type names

    master

    If a DBF field type character is not a valid Python identifier (e.g., + for autoincrement), you must name your parsing method using its hexadecimal ASCII value prefixed with parse.

    For example, to implement a parser for +, name the method parse3F (since ord('+') is 0x3F).

    You can generate the method name programmatically using:

    'parse' + format(ord('?'), 'x').upper()

    Replace '?' with your target field type character.

    >>> 'parse' + format(ord('?'), 'x').upper()
    'parse3F'
  6. Export DBF data to a Pandas DataFrame

    master

    To convert a DBF table into a Pandas DataFrame, you must wrap the DBF object in iter(). Without iter(), Pandas will not recognize that the table is iterable.

    Note: Pandas creates an internal list of all records before converting them to a DataFrame, which means the entire dataset will be loaded into memory.

    import pandas as pd
    from dbfread import DBF
    
    df = pd.DataFrame(iter(DBF('example.dbf')))
    print(df)
  7. Migrate from deprecated dbfread.open() and dbfread.read() to the DBF class

    master

    As of version 2.0, dbfread.open() and dbfread.read() are deprecated and scheduled for removal in version 2.2. To ensure future compatibility, you should switch to using the DBF class directly.

    Note that open() and read() currently return a DeprecatedDBF object (which is a subclass of both DBF and list for backward compatibility), but you should transition to DBF to avoid breaking changes when they are removed.

    # Deprecated approach
    table = dbfread.read('people.dbf')
    
    # Recommended approach
    from dbfread import DBF
    table = DBF('people.dbf', load=True)
  8. Read DBF files with dbfread

    master

    Use the DBF class to read DBF files (used by dBase, Visual FoxPro, and FoxBase+). The library iterates through the file and returns each record as a Python dictionary containing native Python data types (e.g., datetime.date for date fields). This is ideal for batch processing and one-off scripts.

    from dbfread import DBF
    
    for record in DBF('people.dbf'):
        print(record)
    # Output example:
    # {'NAME': 'Alice', 'BIRTHDATE': datetime.date(1987, 3, 1)}
  9. Export DBF data to CSV

    master

    You can export DBF data to a CSV format using Python's standard csv module. This involves iterating over the DBF table object and writing each record to a CSV file.

    import csv
    from dbfread import DBF
    
    for record in DBF('example.dbf'):
        print(record)
    
    # To write to a CSV file using the standard csv module:
    with open('output.csv', 'w', newline='') as f:
        writer = csv.DictWriter(f, fieldnames=['NAME', 'BIRTHDATE'])
        writer.writeheader()
        for record in DBF('example.dbf'):
            writer.writerow(record)
  10. Open and iterate over a DBF file

    master

    To read a DBF file, instantiate the DBF class with the file path. You can iterate over the resulting object to get records as dictionaries (or collections.OrderedDict in older Python versions). Records are returned in the order they appear in the file.

    To access deleted records, use the .deleted attribute. You can also use the with statement for convenience, though the DBF object does not keep files open by default.

    from dbfread import DBF
    
    # Open the table
    table = DBF('people.dbf')
    
    # Iterate over records
    for record in table:
        print(record)
    
    # Access deleted records
    for record in table.deleted:
        print(record)
    
    # Get total record count
    print(len(table))
    
    # Using context manager
    with DBF('people.dbf') as table:
        for record in table:
            print(record)
  11. Migrate from deprecated open() and read() APIs

    master

    As of version 2.0, dbfread.open() and dbfread.read() are deprecated and will be removed in version 2.2. You should migrate to using the DBF class directly.

    Note that the DBF class is no longer a subclass of list. While iteration and counting still work, you must use the .records attribute for list-like operations such as indexing.

    # Old way (deprecated):
    # table = dbfread.read('people.dbf')
    # print(table[1])
    
    # New way:
    from dbfread import DBF
    table = DBF('people.dbf', load=True)
    print(table.records[1])