EFCore.BulkExtensions

repository·master·Indexed 26 days ago

https://github.com/borisdj/efcore.bulkextensions

A library providing high-performance bulk CRUD operations (Insert, Update, Delete, Upsert, Sync, Read) for Entity Framework Core. It bypasses standard EF Core overhead using optimized database-specific protocols such as SqlBulkCopy for SQL Server and COPY BINARY for PostgreSQL. Supports SQL Server, PostgreSQL, MySQL, Oracle, and SQLite via provider-specific NuGet packages. Includes features like BulkSaveChanges for tracked entities, BulkRead for large lists, and BulkInsertOrUpdateOrDelete for table synchronization.

Tokens
4.5K
Snippets
16
Records
27
Agent score
38%

What's inside EFCore.BulkExtensions

  1. Handle Identity columns with SetOutputIdentity

    master

    When performing BulkInsert on tables with Identity columns (e.g., int autoincrement), set SetOutputIdentity = true to have the database-generated IDs updated back into your entity list. This is essential when inserting related parent-child entities where the child needs the parent's new ID as a Foreign Key.

    // Recommended pattern for parent-child relationships
    using (var transaction = context.Database.BeginTransaction())
    {
        context.BulkInsert(entities, new BulkConfig { SetOutputIdentity = true });
        foreach (var entity in entities) {
            foreach (var subEntity in entity.ItemHistories) {
                subEntity.ItemId = entity.ItemId; // Sets FK to the generated PK
            }
            subEntities.AddRange(entity.ItemHistories);
        }
        context.BulkInsert(subEntities);
        transaction.Commit();
    }
  2. Handle TPH (Table-Per-Hierarchy) inheritance

    master

    For TPH models where the Discriminator is a Shadow Property, you must first add the entities to the DbSet so the discriminator value is set before calling the bulk operation.

    Example:

    public class Student : Person { ... }
    // Add to context so Shadow property 'Discriminator' is populated
    context.Students.AddRange(entities);
    context.BulkInsert(entities);
  3. Optimize performance for large data sets

    master
    Bulk operations involve overhead because they create and drop temporary tables. For optimal performance, it is recommended to use Bulk operations only for data sets containing more than 1,000 records. For smaller sets, standard EF Core operations may be more efficient.
  4. Configure MySQL and SQLite for Bulk operations

    master

    Certain databases require additional configuration to support bulk operations:

    • MySQL: You may need to enable local_infile by executing the SQL command: SET GLOBAL local_infile = true;.
    • SQLite: Requires the SQLitePCLRaw.bundle_e_sqlite3 package and a call to SQLitePCL.Batteries.Init().
  5. Install EFCore.BulkExtensions via NuGet

    master

    You can install the main package which supports all databases, or install provider-specific packages to reduce your project's footprint.

    To install the main package using the Package Manager Console, run: Install-Package EFCore.BulkExtensions

    Provider-specific packages follow the naming convention EFCore.BulkExtensions.[Provider] (e.g., EFCore.BulkExtensions.SqlServer, EFCore.BulkExtensions.PostgreSql, EFCore.BulkExtensions.MySql, EFCore.BulkExtensions.Oracle, or EFCore.BulkExtensions.Sqlite).

    Install-Package EFCore.BulkExtensions
  6. Avoid lock escalation in SQL Server using Batch iteration

    master

    To prevent lock escalation in SQL Server when performing large batch operations, use a chunk-based iteration pattern with Take(chunkSize).

    // Batch iteration (useful in same cases to avoid lock escalation)
    do {
        rowsAffected = query.Take(chunkSize).BatchDelete();
    } while (rowsAffected >= chunkSize);
  7. Select the correct EFCore.BulkExtensions NuGet package

    master

    EFCore.BulkExtensions is organized into provider-specific packages to minimize dependencies. You can either install the main package which includes all providers, or install only the specific provider package required for your database.

    • EFCore.BulkExtensions: The main package containing all provider implementations (SqlServer, PostgreSql, MySql, Oracle, Sqlite).
    • EFCore.BulkExtensions.SqlServer: For SQL Server.
    • EFCore.BulkExtensions.PostgreSql: For PostgreSQL.
    • EFCore.BulkExtensions.MySql: For MySQL.
    • EFCore.BulkExtensions.Oracle: For Oracle.
    • EFCore.BulkExtensions.Sqlite: For SQLite.

    All provider-specific packages depend on EFCore.BulkExtensions.Core.

  8. Set up MySQL via Docker Compose

    master

    To run a MySQL instance, use the mysql image. Note that the command includes --local-infile=1 which is often required for bulk data operations. The default configuration uses:

    • Database: bulk
    • Root Password: MySQL22
    • Port: 3306
    mysql:
      image: mysql
      environment:
        - MYSQL_DATABASE=bulk
        - MYSQL_ROOT_PASSWORD=MySQL22
      ports:
        - "3306:3306"
      volumes:
        - mysql_data:/var/lib/mysql_data
      command: --local-infile=1
  9. Set up PostgreSQL/PostGIS via Docker Compose

    master

    To run a PostgreSQL instance with PostGIS support for testing, use the postgis/postgis image. The default configuration uses the following credentials:

    • User: postgres
    • Password: Postgres22
    • Database: bulk
    • Port: 5432
    postgres:
      image: "postgis/postgis"
      ports:
        - "5432:5432"
      environment:
        - POSTGRES_USER=postgres
        - POSTGRES_PASSWORD=Postgres22
        - POSTGRES_DB=bulk
      volumes:
        - postgis_data:/var/lib/postgresql/data
  10. Execute multiple Bulk operations in a single transaction

    master

    By default, each Bulk operation is a separate transaction and is automatically committed. To group multiple Bulk operations into a single atomic transaction, use context.Database.BeginTransaction().

    using (var transaction = context.Database.BeginTransaction())
    {
        context.BulkInsert(entities1List);
        context.BulkInsert(entities2List);
        transaction.Commit();
    }
    
    // or with C# 8.0+ syntax
    using var transaction = context.Database.BeginTransaction();
    context.BulkInsert(entities1List);
    context.BulkInsert(entities2List);
    transaction.Commit();